Changing Assessment Cultures: The Northern Ireland Experience
(Queens University, Belfast)
Paper presented at the British Educational Research Association Annual Conference, University of Sussex at Brighton, September 2-5 1999
This paper describes aspects of the assessment system for Northern Ireland schools, including approaches to end-of-key stage testing, baseline assessment, moderation and target setting. Broadly speaking, assessment procedures and methods have followed the QCA/SCAA lead for some years but some differences have crept in. A selection of the similarities and differences are discussed and likely developments in assessment over the next several years are considered.
For almost thirty years education developments in Northern Ireland have followed a London lead, primarily because there has been direct rule since the suspension of the Northern Ireland parliament in the early 1970s. Successive ministers of education, and ministers with other portfolios such as health or trade and industry, have been appointed by London with the simple remit: do as we do. Whether or not the various initiatives have been taken on board seems largely to have been down to the will and inclination of the local administration (the Department of Education for Northern Ireland, DENI) to counter-advise the minister or, sometimes, the strength of feeling in the local constituencies: the education and library boards (i.e. the LEAs), schools, teachers and parents. Most times, DENI promotes the simple position: "We must adopt this change to ensure our education system remains compatible with England and Wales" - an imperative that has not troubled Scottish education unduly! Hence we have a Northern Ireland version of the National Curriculum, major changes in teacher education, parallel changes in how we cater for special needs etc. On other occasions, however, it seems that ministers accept a polite: "No, thank you" or perhaps, from some quarters, a: "We most certainly will not!" Moves to introduce comprehensive schools, for example, or an OFSTED-type re-organization of the Inspectorate have therefore remained on distant shores.
A crucial tactic in dealing with all imported educational developments has been the staggered introduction; a "lets wait and see what the pitfalls are and then we can adopt the system that works". In this manner various initiatives (e.g. GCSE, the Northern Ireland Curriculum and more recently baseline assessment) have deliberately lagged a year or two behind changes in England and Wales with a view to picking up best practice and avoiding the worst of any teething problems. As will be argued later, there is an element of a somewhat blind, follow my leader approach, which also assimilates the bad practice that is systemic in some initiatives.
The agency charged with providing an assessment system in the compulsory education sector - external examinations at all levels, moderation support, the 11+ selection system etc. - is the Northern Ireland Curriculum Council for Examinations and Assessment, (NI)CCEA. They also advise the Government on curriculum and assessment policy and in this respect much of their work derives directly from initiatives and developments in England: "In practical terms the Council is required to ensure equivalence in educational standards with England in Key Stage Assessment and examinations ..." (CCEA 1999c, p30). The most significant case in point is the Northern Ireland Curriculum (NIC), introduced in 1990. Almost identical in shape and content, the NIC does however differ from the National Curriculum (NC) in some minor and one or two significant ways. As examples of the former, the NC subject area Design & Technology was named Technology & Design in Northern Ireland, and Speaking & Listening became Talking & Listening - viewed by some at the time as somewhat unsubtle and unhelpful attempts to maintain a semblance of autonomy! More significantly, information technology did not feature as a formal part of Technology & Design but as a cross-curricular theme - seen not as a discrete subject in Northern Ireland but as a pervasive cross-curricular theme to be accommodated in all aspects of a pupils schooling. The cross-curricular themes, Education for Mutual Understanding and Cultural Heritage, also edged out Citizenship (presently making a comeback under a recent curriculum review) but this difference is more obviously due to local considerations.
In assessment of the new curriculum there was less room to manoeuvre. The 10-level scale and its attendant statements of attainment, attainment targets and profile components therefore created the same havoc in Northern Ireland as elsewhere in England and Wales, despite any time lag used. The same struggles ensued around the concepts of rigour (involving reliability), purpose (involving validity) and standardization (involving comparability and generalizability) - to greater or lesser extents relating to problems with the same concept and process: teacher assessment. This has evolved in the same way in Northern Ireland with big T A - statutory assessment procedures involving the teachers judgement of the pupils performance for the purpose of grading - all but eclipsing small t a - the teachers involvement in day-to-day classroom assessment for the purpose of assisting learning. Before dealing with statutory teacher assessment per se, it is worth looking at two recently developing areas: baseline assessment and target setting.
The wait-and-see time lag has been much more pronounced in the introduction of baseline assessment, so much so that when presented with a programme that included two year-long pilot studies, the then minister of education was moved to describe it as "protracted" (CCEA 1998a, p 4). Nevertheless he was "content" to accept CCEAs plans to implement a statutory baseline assessment at a projected time (September 2000) when all English primary schools would have been doing it for two years.
Much of the proposed system finds resonance with the English schemes but there are a number of interesting and, in some cases, crucial differences. There is only one accredited system proposed for Northern Ireland (c.f. the 90 accredited schemes in England which may or may not adopt the QCA baseline assessment scales) but the guidance material (CCEA 1997) takes a similar approach to that of the desirable outcomes set out by SCAA (1997). As might be expected, there are differences in nomenclature and structure.
The proposed Northern Ireland baseline assessment system is structured in three Progress in Learning sections:
Personal, Social and Emotional Development (PSED);
Language Development; and
Early Mathematical Experiences.
Each section is divided into three stages of progression with most pupils considered likely to be in the second stage. Both the Language Development and Early Mathematical Experiences sections each have three sub-sections as set out in Table 1:
Table 1 Northern Ireland Baseline Assessment Areas in Addition to Personal, Social and Emotional Development
|Language Development||Early Mathematical Experiences|
|Talking and Listening Skills||Talking about Mathematical Ideas|
|Early Reading Skills||Patterns and Relationships|
|Early Writing Skills||Early Number Concepts|
PSED has several descriptions of pupil behaviour constituting a typical profile of pupils at each of the same three stages of progression. The emotional behaviour descriptors of the PSED represent an interesting difference from the Personal and Social Development (PSD) of the basic accredited system in England.
A more crucial difference in the two systems, however, is the manner of arriving at a judgement of the baseline assessment of the child. In the basic accredited system in England, the eight assessment scales cover much of the same ground as that in Northern Ireland: Reading (three scales), Mathematics (two scales), Writing, Speaking & Listening and PSD. Teachers in both systems are encouraged to use all available sources of information in arriving at a judgement e.g. parents, pre-school personnel, classroom assistants etc. In the Northern Ireland system it is proposed that judgements be made on a best fit basis against the profile descriptions while in England, pilot studies of the accredited schemes have led to a rejection of best-fit approaches as overly complex, and Yes-No checklists as insufficiently informative. The resulting scheme in England therefore uses four individual criteria that are progressive within each scale, with the exception of PSD in which each criterion is independent. This leads to a profile of eight scale scores or an aggregate (max = 32), the latter to be used for school-level value-added calculations at the end of key stage 1. Although the Northern Ireland scheme has started out as a series of descriptive profiles, their use in value-added and target setting contexts is also anticipated:
"In order to provide a measure for calculating value added, at the end of Key Stage 1 scores should be attributed to the assessments made in each curricular area in baseline assessment. There should be a scale of up to 9 for each area and a maximum aggregate score of 27." (CCEA 1998a, p 19).
The justification for aggregating the individuals profile appears to have been back-extrapolated from a school baseline measure: "Recent research carried out in England for the School Curriculum and Assessment Authority ... [which indicated] that the most appropriate way of providing a baseline for measuring the value added by schools ... is to use an aggregated pupil score on a numerical scale" (CCEA 1998a, p 18). As is the case in England, no mention is made of any weightings, measurement errors etc. and the implications of mis-classification, if any, are not addressed.
Reporting to parents is more or less the same in each scheme. Schools must provide parents with an opportunity to discuss the baseline assessment, with provision made for parents who exercise their right to have the assessment report in writing. In Northern Ireland the proposal is that schools will discuss with the parents the profile that fits their child in each of the seven areas. Care must be taken not to give the impression that children could already be failing so early in their formal schooling. All respondents to a consultation exercise carried out by CCEA rejected the notion of attributing a single score to individual children (CCEA 1998a).
Since September 1998, schools in Northern Ireland have had to set targets for improvement as part of their development planning. Table 2 summarizes the key stage 1, 2 and 3 targets that have been proposed by DENI (1998a & 1998b) for achievement in literacy and numeracy by 2002. The targets have been disseminated as NIC levels in English and mathematics.
Table 2 Northern Ireland Targets for Literacy and Numeracy [key: % of pupils in 2002, (% in 1998)]
|Subject||Level 2||Level 3||Level 4||Level 5||Level 6|
Key Stage 1
|Maths||100 (92)||35 (25)|
|English||100 (91)||35 (25)|
Key Stage 2
|Maths||80 (67)||35 (28)|
|English||80 (62)||25 (14)|
Key Stage 3 - Non-Selective Secondary
|Maths||75 (60)||33 (29)|
|English||60 (45)||26 (20)|
Key Stage 3 - Selective Secondary (Grammar)
|Maths||100 (99)||100 (94)|
|English||100 (98)||90 (81)|
Key Stage 3 - All Secondary
|Maths||85 (74)||55 (51)|
|English||75 (64)||50 (42)|
The DENI documentation clearly advises that the targets are for the whole system, and not necessarily for individual schools, but examination of the table will show that the attenuation between current and target values for grammar schools could be problematic. For example, a grammar school with 100 pupils in Year 10(1), and with 99% of their pupils at level 5 in mathematics (the current average), will have to meet the target by keeping an extra careful eye on one or two children!
What is more seriously problematic, however, is the underlying weakness in attributing the quality of a standard to NC/NIC levels. Here is an example of the blind aspect of follow my leader. England chooses to use complex level descriptions as standards and Northern Ireland follows suit; bringing to the target setting process the same old problems of differences in subject interpretation and inconsistency within and between schools. Even if the levels are established by external tests, their interval scaling is still widely considered to represent at least two years progression in learning. As such they may have a significant component of measurement error. In Northern Ireland, in key stages 1 and 2, the levels are awarded through teacher assessment only; albeit supported through calibration with CCEA-provided Assessment Units (AUs) and a moderation process once every three years. The contribution of error to the judgement of the level (with one judge: the teacher) must therefore be potentially greater. No account of measurement error appears to be taken in the English system and none is apparent in the Northern Ireland system. Should not, then, the targets be expressed in more tolerant ranges? For example is not 20-40% more realistic than a definitive 35% for the proportion of key stage 1 pupils reaching level 3 in mathematics by the end of the key stage? Politically, of course, this would just not do!
Statutory Assessment Arrangements
Statutory assessment in Northern Ireland is significantly different from England at key stages 1 and 2, inasmuch as it is based wholly on teacher assessment. Teachers are required to assess each pupil in each attainment target in mathematics and English (or Irish in Irish-medium schools) and to record the level awarded after best-fit consideration of all of the evidence available. In a cautionary echo of the evidence debacle of the early 1990s, teachers are advised that "... elaborate arrangements for recording assessments and retaining evidence are neither required by the Council nor necessary to satisfy DENI inspection requirements" (CCEA 1998b, p 5). Key stage 3 is more similar to the English system with teacher assessment and externally marked tests providing a two component report to parents (CCEA 1999a). There is no moderation of the teacher assigned levels but internal standardization is recommended as good practice. Both components nevertheless "... have equal status and provide complementary information about pupils attainment" (CCEA 1999b, p 4). The assessments cover science as well as mathematics and English (or Irish in Irish-medium schools).
High on validity, on the basis that teachers in-depth knowledge of their pupils places them in a unique position to judge their progress, teacher assessment suffers from problems of what is often termed rigour. Not rigour in the sense of how conscientiously or otherwise it is carried out, but rigour in the various technical senses of reliability including accuracy, consistency and comparability between judges. For key stages 1 and 2 (but not 3), the problem is addressed in two ways in Northern Ireland: by the use of Assessment Units to calibrate teachers judgements and by a process of Quality Assurance Moderation.
Assessment Units, AUs
Assessment Units are short tests or tasks designed to be completed by the pupils and then marked, with the assistance of a marking scheme, by the teacher. In theory, the teacher will have formed a view on each pupils level in the particular attainment target from all of the sources of evidence available to them e.g. classwork, homework, class tests etc. The AU then serves to corroborate the teachers judgement or to assist them in fine-tuning it. In practice, the following cameo, paraphrased from a teachers account of their experience in one school, might be more prevalent that would be wished:
The period for statutory assessment and mandatory use of AUs began on January 1 and was to be completed, with a level assigned to each pupil, by May 8. Teachers were advised to arrive at the level, which best fitted each pupils performance in each target, by drawing upon written, practical and oral work in the classroom. CCEA required that levels in Reading, Writing, Number and one other mathematics attainment target be confirmed using AUs, the results of all assessments to be recorded in the Class Assessment Record. By Easter it was clear that pressure of other work had caused little in the way of level assignment to be achieved. After Easter there were barely three weeks to complete the assessment; 18 weeks had contracted to three. Despite it being clear that AUs were to be used to help confirm judgements, the teachers spoke openly of using them to establish the levels. In some cases the AUs were the only evidence of attainment.
The AUs themselves are subject to audit; schools being randomly selected to check the accuracy of their marking of the AUs. The audit report for 1998 (CCEA 1998c) indicated that the "great majority" (p 9) of units in mathematics and English were marked accurately but the teachers in the school above either genuinely ran out of time for the prior assessments that were required or delayed their assessments because they were not confident in their own judgements. Another alternative is that they quite simply used the AUs as a form of external standardized tests - which in some respects they are.
Quality Assurance Moderation
Black (1998) has said that the four principles upon which a (National Curriculum) assessment system should be based are: "... criterion referencing, progression, formative and moderated [sic]" (p 58). Assessment of key stage 1 and 2 in Northern Ireland is referenced to established criteria, which are based on progression in learning. Big T A has arguably reduced the incidence of formative small t a but moderation of teacher assessment (T A) has been formally established in Northern Ireland and is known as Quality Assurance Moderation.
Quality assurance moderation takes the form of requiring schools to provide, once in every three years, combined key stage 1 and 2 portfolios for mathematics and English each comprising six pieces of marked pupil work at each of the levels 2, 4 and 5. The English portfolio must have three pieces of work for both Reading and Writing at each level and the mathematics portfolio similarly requires three pieces each for two pre-selected attainment targets. In addition, input from Council moderators for agreement trials is available. The portfolios are either endorsed as indicating agreement with the Councils required standards or returned with feedback detailing any divergence from them. In the case of a portfolio not being endorsed, the school is required to make the necessary "adjustment of standards" (CCEA 1998b, p 4) and to resubmit their portfolio in the following year. This is a comprehensive approach to promoting rigour in the use of teacher assessment but again a school-based cameo, also paraphrased from one teachers account and not by any means argued as typical, suggests that its operational efficacy can be problematic:
An afternoon session of a Baker day was set aside for internal standardization of the levelling of pupils work in English and mathematics. Two collections of work had been sent to the Council which agreed to send four moderators to facilitate the session. During the session, the moderators circulated from group to group, contributing to debates about the correct level for each piece of work and sometimes giving a definitive decision. While some teachers accepted the moderators evaluations, others remained dissatisfied with them and continued to argue their point in their respective groups. Many of the debates on the English pieces remained unresolved by the end of the session but agreement in the mathematics session proved much easier to achieve. Two weeks later the school began an internal standardization on four folders of work from which the mathematics and English portfolios were to be drawn for moderation by the Council. Over a period of four weeks the folders were rotated between the groups concerned, without them knowing the previous groups deliberations. The difficulties encountered far exceeded expectations, particularly in English. In one case a piece of writing work was judged by one group as level 2 and by another as level 4. Eventually agreement was reached, in some cases reluctantly, in the selection of work for the two portfolios and they were sent off. Considering the comparative ease with which the mathematics portfolio was constructed, the staff were subsequently astounded to have their mathematics portfolio returned as requiring adjustments at each of the three levels while the English portfolio levels were confirmed.
It would appear that the twin fixes of AUs and moderation have until recently been considered sufficient to underpin school comparisons. However, in a recent survey (summarized in CCEA 1999d), teachers at both key stages rejected teacher assessment as not being "sufficiently rigorous" as the basis for comparing standards across schools (pp 32 & 34). The same survey recorded key stage 1 and 2 teachers appreciation of the role of the AUs in calibrating judgements but also noted their preference for standardized tests as being more useful than statutory assessments. Level descriptions were also endorsed as being useful for teachers but their wording and numerical scoring were felt to be unhelpful to parents or for indicating a pupils individual progress in relation to the peer group respectively.
Future Developments in Assessment in Northern Ireland
The prospect of a return to local governance of education is awaited throughout the Northern Ireland system with some anticipation. It has been argued in this paper that for several decades now, there has been very much a follow my leader approach. This will not dissipate completely, nor should it. Much has been learned from our neighbours and there is clearly much more to be learned. One difference will certainly be the exercise of choice in what happens in Northern Ireland education.
CCEA (1999c) has not stayed idle as the changes approach and indeed have proposed a radical review, over the next two years, of all aspects of current assessment provision. This is to include what learning is assessed, how and when this is carried out and to what purpose: formative, diagnostic and summative etc. While acknowledging the benefits of promoting good assessment practice through statutory teacher assessment, supported by moderation, they also identify a growing preference among schools for pencil and paper tests that are marked externally; a preference that they perhaps intend to serve. They suggest that the growing distrust of teacher assessment, which they have perceived, is born out of the increased competition between schools and the "culture of public accountability measured by assessment and examination success" (p 30). Leaving these relatively hackneyed issues to one side, however, they also identify proposals to develop alternative modes and styles of assessment that will, for example, allow for:
- the more effective assessment of a range of thinking skills;
- the opportunity to contribute to group problem-solving and team work; and
- the assessment of interpersonal negotiating and conflict management skills.
These are interesting proposals and represent no easy task! Indeed to conclude in CCEAs own words, they will require: "Considerable political will, imagination and resources ... to devise an effective, assessment system that assesses what is valued as opposed to valuing what is assessed" (p 31).
Black, P. (1998) Learning, league tables and national assessment: opportunity lost or hope deferred? Oxford Review of Education, 24 (1) 57-68
CCEA (1997) Curricular Guidance for Pre-School Education. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office
CCEA (1998a) The Ministers Response to the Councils Advice on Baseline Assessment in Northern Ireland. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 085 1
CCEA (1998b) Assessment Arrangements in Keys Stages 1 and 2. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 0 337 04325 6
CCEA (1998c) Evaluation Report: Statutory Assessment 1997/98. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 110 6
CCEA (1999a) Assessment Arrangements in Key Stage 3. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 192 0
CCEA (1999b) Assessment Arrangements in Key Stage 3: Detailed Procedures. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 193 9
CCEA (1999c) Developing the Northern Ireland Curriculum to Meet the Needs of Young People, Society and the Economy in the 21st Century. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 195 5
CCEA (1999d) Key Messages from the Curriculum 21 Conferences and the Curriculum Monitoring Programme 1998. Northern Ireland Council for Curriculum, Examinations and Assessment, Belfast: The Stationery Office ISBN 1 85885 196 3
DENI (1998a) Target Setting: Guidance for Post-Primary Schools. School Improvement, the Northern Ireland Programme. Department of Education, Northern Ireland Belfast: The Stationery Office
DENI (1998b) Target Setting: Guidance for Post-Primary Schools. School Improvement, the Northern Ireland Programme. Department of Education, Northern Ireland Belfast: The Stationery Office
SCAA (1997) Looking at Childrens Learning: Desirable Outcomes for Childrens Learning on Entry to Compulsory Education. School Curriculum and Assessment Authority, London: The Stationery Office
1. Year 10 is the Northern Ireland equivalent to Year 9 at the end of key stage 2 in England, owing to a younger school entry age to Year 1 for most primary school children. Transfer to secondary schools occurs at the end of Year 7, instead of Year 6, when most of the children are in their 11th year.