Stephen Gorard, Jane Salisbury, and Gareth Rees

School of Social Sciences
21 Senghennydd Road
Cardiff University
CF2 4YG
01222-875113

email: gorard@cardiff.ac.uk

There are at least two methods of calculating achievement gaps (between groups of students in education) in common current usage, similar to those used to calculate social segregation and mobility. Each method clearly seems valid to its proponents, yet their results in practice are radically different, and often contradictory. This brief paper considers both of these methods and some related problems in the calculation of achievement gaps, in an attempt to resolve the contradiction. The issue is a simple one, but one with significant implications for social researchers, as well as commentators in many areas of public policy using similar indicators of performance.

As a result of large-scale analysis undertaken for the Qualifications, Curriculum and Assessment Authority for Wales, the standard picture of the underachievement of boys at school has become more complex (Gorard et al. 1999a). It is now clear that underachievement, if and where it exists, is not uniform in nature, varying as it does between regions, years, levels and modes of assessment, and applying only to some subjects. It is also clear that in such combinations where 'gender gaps' appear the gaps are concentrated at higher levels of achievement, and that differences between boys and girls on aggregate measures are decreasing over time.

When examined at an all-Wales level, using total subject entries, it is clear that girls perform slightly better than boys in the system of statutory assessment and examination at KS4 (Table 1). For example, girls tend to enter more, and more varied, subjects at GCSE, achieving higher grades overall. At A level, where students exercise much greater choice of subjects, entry gaps tend to be larger, whilst the achievement gaps tend to be smaller than in GCSE. In general, gender appears to play less of a role in attainment at A level. When broken down into subject groups, achievement gaps are largest in English (and Welsh) at KS1 to KS4. There are also significant achievement gaps in some other subject groups. These gaps appear year after year, and they are nearly always in favour of girls. The exceptions to this pattern are Mathematics and Sciences which constitute the majority of core subjects, where there are no systematic differences at any age between the performance of boys and girls.

There are also no systematic differences at any age between the performance of boys and girls at the lowest level of any measure of attainment, such as Level 1 KS1 or grade G GCSE. The overall conclusion is therefore that the system assesses girls and boys equally at the lowest levels. This finding is in direct contradiction to theories that the achievement gap is primarily a problem at lower levels of attainment and among lower ability or 'demotivated' boys. The achievement gap in favour of girls (in subjects where it exists) is actually largest at the highest levels of attainment (Table 2). In general, the size of the achievement gap gets larger at successive attainment levels. The gap in favour of girls at middle levels of attainment in subjects where it exists, such as grade C in GCSE English, has been relatively constant for the past six years. In contrast to reports of a growing overall achievement gap, even in the subjects where girls' superior performance is most marked, the differential attainment of boys and girls remains static or even reduces over time when considered at the benchmark levels, such as level 2 at KS1 or grade C in GCSE.

The achievement gap between boys and girls in terms of aggregate measures, such as the percentage attaining five or more GCSEs at grades A* to C, has declined since 1992 (Table 3). This is in contrast to some reports of a growing achievement gap, and is chiefly explained by the fact that some previous analysts have mistakenly used the percentage point difference between boys and girls as a measure of the gap in achievement, despite an annual growth in the proportion of the age cohort achieving each grade. These findings provide an important corrective to many previous accounts of boys' 'under-achievement'. Of course, it remains a matter of concern that, in general terms, boys are performing less well than girls in any subjects (and vice versa), and also if any students of either gender are underachieving. Nevertheless, in terms of the pattern of achievement gaps identified here, it is important that the scale and nature of this 'under-achievement' is clearly understood in order for research on the reasons for the gaps to be valid, and for appropriate policies to be drawn up with it.

All of these findings have implications for the conduct of future research to explain differential performance. If boys are not underachieving at low levels and grades in any subject at any age, and not at all in the majority of core subjects, and where boys are gaining lower grades than girls they are catching up over time, then a lot of existing work on gender and education is trying to explain, and many policies and action research projects are trying to ameliorate, a phenomenon that does not actually exist. The standard 'crisis' account has failing boys, growing gaps, and worsening with age as its pillars. Therefore, before we can begin the detailed fieldwork necessary to explain our findings - indeed perhaps before we can even persuade some referees that our findings are valid and generalisable since they see journalists, politicians and even academics present the crisis version almost daily - we need to resolve what we have termed the 'paradox of achievement gaps'.

The calculation and discussion of achievement gaps between different sub-groups of students ('differential attainment') has become common among policy-makers, the media, and academics. An 'achievement gap' is an index of the difference in an educational indicator (such as an examination pass rate), between two groups (such as males and females). In addition to patterns of differential attainment by gender, recent concern has also been expressed over differences in examination performance by ethnicity, by social class, and by the 'best' and 'worst' performing schools. The concerns expressed in each case derive primarily from growth in these gaps over time.

Accounts generally use one of two substantially different methods of calculating differential attainment over time. The first and most common method uses percentage points as a form of 'common currency'. Thus, if 30% of boys and 40% of girls gain a C grade in Maths GCSE in one year, and 35% of boys and 46% of girls gain the equivalent a year later, the improvement among girls is said to be greater, in the way that six (46-40) is greater than five (35-30). This is justified by its advocates since percentages are, in themselves, proportionate figures. If true, it would mean that girls were now even further 'ahead' of boys than in the previous years. Thus, the gender gap has grown.

The second general method calculates the change over time in proportion to the figures that are changing. This approach is advocated by Charles Newbould and Elizabeth Gray in an EOC study of gendered attainment (Arnot et al. 1996, see also Gorard et al. 1999b). For them, an achievement gap is the difference in attainment between boys and girls, divided by the number of boys and girls at that level of attainment. More formally, the entry gap for an assessment is defined as the difference between the entries for girls and boys relative to the total entries.

where GE = number of girls entered; and BE = number of boys entered (or in the age cohort)

The achievement gap for each outcome is defined as the difference between the performances of boys and girls, relative to the performance of all entries, minus the entry gap.

where GP = the number of girls achieving that grade or better; BP = the number of boys achieving that grade or better.

Now the interesting thing about these two common methods is that they give totally different results from the same raw data. For example, Gibson and Asthana (1999) claim that the gap in terms of GCSE performance between the top 10% and the bottom 10% of English schools has grown significantly from 1994 to 1998. Their figures are reproduced in Table 4. This shows the proportion of students attaining five or more GCSEs at grade C or above (the official benchmark), for both the best and worst attaining schools in England. It is clear that the top 10% of schools has increased its benchmark by a larger number of percentage points than the bottom 10%. The authors conclude that schools are becoming more socially segregated over time, since 'within local markets, the evidence is clear that high-performing schools both improve their GCSE performance fastest and draw to themselves the most socially-advantaged pupils' (in Budge 1999, p.3).

This conclusion would be supported by a host of other commentators using the same method (including Robinson and Oppenheim 1998, and Chris Woodhead, in the Times Educational Supplement 12/6/98, p.5). Similar conclusions using the same method have been drawn about widening gaps between social classes (Bentley 1998), between the attainment of boys and girls (Stephen Byers, in Carvel 1998, Bright 1998, Independent 1998), between the performance of ethnic groups (Gillborn and Gipps 1996), and between the results of children from professional and unemployed families (Drew et al., in Slater et al. 1999).

The second method, using the same figures, might produce a result like Table 5. Although the difference between the deciles grows larger in percentage points over time, this difference grows less quickly than the scores of the deciles themselves. On this analysis, the achievement gaps are getting smaller over time. This finding is confirmed by the figures in the last column showing the relative improvement of the two groups. The rate of improvement for the lowest ranked group is clearly the largest (and it may be significant that the rate for the intervening eight deciles is also 1.09, see below). The bottom decile would, in theory at least, eventually catch up with the top decile (Gorard 1999a). The same reanalysis can be done in each of the examples above to show that the gaps between schools, sectors, genders, ethnic groups, and classes are getting smaller over time. This would be the exact opposite in each case to the published conclusions.

To summarise the position so far: using the most popular method of comparing groups over time there appears to be a crisis in British education. Differences between social groups, in terms of examination results expressed in percentage points, are increasing over time and so education is becoming increasingly polarised by gender, class, ethnicity, and income. Using the second method, when these differences are considered in proportion to the figures on which they are based, the opposite trend emerges. Achievement gaps between groups of students defined by gender, ethnicity, class and income actually appear to be declining. Education is becoming less polarised over time. This is the 'paradox of achievement gaps'. Both methods are used in different studies. Both have been extensively published and peer-reviewed. Some writers have even used the equivalent of both methods in the same study (e.g. Levacic et al. 1998, Lauder et al. 1999). Surely someone has to decide once and for all which method to use, as they are not simple variants of one another?

Very similar analyses also occur in social science more generally, and similar problems have arisen in health research (Everitt and Smith 1979), in studies of socio-economic stratification and urban geography (Lieberson 1981), in social mobility work (Erikson and Goldthorpe 1991), and in predictions of educational pathways (Gorard et al. 1999c). Results are disputed when an alternative method of analysis produces contradictory findings. Some of these debates are still unresolved, dating back to what Lieberson (1981) calls the 'index wars' of the 1940s and 1950s. In each case the major dispute is between findings obtained using absolute rates ('additive' models) and those using relative rates ('multiplicative' models).

Absolute rates are expressed in simple percentage terms, while relative rates (odds ratios) are margin-insensitive in that they remain unaltered by multiplication of the rows or columns (as might happen over time for example). This difference is visible in changes to the class structure and changes in social mobility. In Table 6, 25% of those in the middle class are of working class origin, whereas in Table 7 the equivalent figure is 40% (from Marshall et al. 1997, pp. 199-200). However, this cannot be interpreted as evidence that Society B is more open than Society A as the percentages do not take into account the differences in class structure between Societies A and B, nor their changes over time ('structural differences').

Relative rates are calculated as odds ratios [(a/c)/(b/d)], cross-product ratios [ac/bd], or disparity ratios [a/(a+c)/b/(b+d)]. Disparity ratios are identical to the segregation ratios used by Gorard and Fitz (1998, 1999). Odds ratios estimate comparative mobility changes regardless of changes in the relative size of classes, and have the practical advantage of being easier to use with loglinear analysis (Gilbert 1981, Goldthorpe et al. 1987, Gorard et al. 1999d). 'From the point of view of social justice... this is of course both crucial and convenient, since our interest lies precisely in determining the comparative chances of mobility and immobility of those born into different social classes - rather than documenting mobility chances as such' (Marshall et al. 1997, p.193 ). The cross-product ratio for Table 6 is 9, and for Table 7 it is also 9. This finding suggests that social mobility is at the same level in each society, despite the differences in class structure between them.

Some previous work has confounded changes in social fluidity with changes in the class structure. Nevertheless, disagreement about the significance of absolute and relative mobility rates continues (e.g. Clark et al. 1990, pp. 277-302). Gilbert (1981) concluded that 'one difficulty with having these two alternative methods of analysis is that they can give very different, and sometimes contradictory results' (p.119). The similarities to the issue concerning achievement gaps are fairly obvious. In each case, different commentators use the same figures to arrive at different conclusions. One group is using additive and the other is using multiplicative models.

Four alternative methods have been mentioned for assessing relationships in a simple two-by-two contingency table. The cross-product (or odds) ratio is commonly used to estimate social mobility, and the segregation (or disparity) ratio (or dissimilarity index) can be used for the same purpose, but is perhaps more generally applicable to the analysis of changes in stratification over time. The achievement gap is used to analyse differential attainment by sub-groups, but is also useful for defining differential access to public services. These three methods are all multiplicative. Percentage points differences have also been used in all of these areas as a rough and ready guide which is easy to calculate. This method is additive in nature.

Despite the differences, there are many similarities between all of the methods and their variants (Darroch 1974). At the limiting case of no relationship (interaction, or change over time), and also for its complete opposite, the methods are identical. Given a two-by-two table of the form:

a	b
c	d

For the cross-product ratio, no change is defined as: ad/bc = 1, equivalent to ad = bc.

For the segregation ratio, no difference is defined as: a/(a+c) / ((a+b)/(a+b+c+d)) = 1, equivalent to a/(a+c) = (a+b)/(a+b+c+d), equivalent to ad = bc.

For the achievement gap, no gap is defined as: (a-b)/(a+b) - ((a+c)-(b+d)) / ((a+c)+(b+d)) = 0, equivalent to (a-b).((a+b)+(c+d)) = (a+b).((a+b)-(c+d)), equivalent to ad = bc.

For the percentage point method, no difference is defined as: 100a/(a+c) - 100b/(b+d) = 0, equivalent to 100a/(a+c) = 100b/(b+d), equivalent to ad = bc.

For other values, although each method gives varying results, all can be used to gauge a pattern or estimate the strength of a relationship. For example, if 100 girls and 100 boys sit an examination, of whom 30 girls and 20 boys achieve a particular grade, the results produced are as in Table 8 (the cross-product ratio is 1.7 etc.). If in a later test 60 of 100 girls and 40 of 100 boys achieve the same grade, the figures from the first and last methods change, while the others remain the same. The method of percentage points suggests that the gap between girls and boys has doubled from Test 1 to Test 2, whereas the cross-product ratio suggests that the gap has increased less dramatically. Both other methods suggest no difference in the differences over time.

Although arguments can and have been made for using either multiplicative or additive methods as measures of association in one table, the chief problem lies in their different results when comparing the patterns in two or more tables. Alone among the methods, using percentage points does not take into account the proportion (a+b)/(c+d) which accounts for rises and falls in the frequency of the phenomenon being observed. Using this method a commentator can have no genuine idea of the significance of the resulting points difference. This can be emphasised in two ways. First: if 1% of men were MPs but 0% of women were, this would be an enormous difference and one that social science commentators would be right to draw attention to. On the other hand, if 75% of men and 76% of women were in paid employment, the difference may be of little account. However, both examples yield a score of 1 point in the additive method, suggesting that this method is fine for a rough guide to the presence or absence of a pattern, but of little value as a measure of achievement gaps. Second: it is surely no coincidence that when changes in percentage points are calculated proportionately to the base frequencies from which they derive, then changes over time between sub-groups are almost constant even in cases where the percentage point method suggests a dramatic widening of differential attainment (e.g. for the top nine deciles presented by Gibson and Asthana 1999).

Of course, it can be argued that none of these methods is appropriate if scores such as the GCSE benchmark are not at least of interval level measurement. Since the number of GCSEs per student in any sizeable group may be approximately normally distributed, it can be argued that changes in the percentage gaining five (or any other arbitrary number) would be less significant near the 50% mark. At the 50% mark, where the distribution is taller, a small movement along the x axis (representing a change in the number of GCSEs per student) would produce a disproportionately large change in the percentage attaining the benchmark. At either end (near 0 and 100%), a much larger change on the x axis would be needed to produce the same effect. In this context it is interesting to note that the supposedly advantaged group in Table I is much nearer the sensitive 50% mark than their supposedly weaker comparator (that is, if followed through to its logical conclusion, this line of reasoning would agree with the multiplicative method in concluding that the improvement in the lower attaining groups was the greater). There are problems with this line of reasoning however, especially when applied to a non-continuous variable such as number of GCSEs per student, but discussion of these will have to remain a subject for the future. If benchmarks are not interval in nature then the crisis account of widening gaps in British education probably cannot be sustained anyway.

At present, the situation is that the specific method of calculation used to assess changes in relative performance over time determines the result obtained. A consensus about the two general methods must be reached quickly by the research community. In terms of social mobility research the preference of most commentators on methodology is clear. Darroch (1974) says 'on balance, the author believes that Hm [the multiplicative definition] is preferable to Ha [the additive definition]' (p.213). Gilbert (1981) suggests that the percentage point difference method can be used 'to assess the association in a percentaged table quickly and roughly' (p.119), but states an overall preference for the relative ratio methods for the kind of practical reasons described above. Ironically, Marshall et al. (1997) are clear in their preference for the relative approach for social mobility studies, but include in their own work a percentage point difference approach to relative changes in educational qualifications over time (p.113).

It is not always clear (from literature review and personal experience) that commentators using the percentage point difference approach are aware that there are other, and better, methods. Without restarting the index wars, it would be wise for this issue to be at least debated in relation to the paradox of achievement gaps. If the multiplicative model is preferred, the consequences would be momentous for much existing research, for the cumulated conclusions of some entire fields of endeavour, the validity of many 'qualitative' studies in related areas, for public research funding priorities, and above all for educational policy. As with the earlier debates about class and stratification, other fields of social science investigation would be affected as well.

Another implication for educational research arising from this study concerns the status of results derived from data for Wales. As has been shown on several occasions, relevant datasets from the principality are often manageable in size, but varied in nature and 'national' in scope. They have therefore been used recently to discern trends in Wales which have predicted what is also happening elsewhere (Gorard 1999b). Although regional differences are important for any analysis (Rees et al. 1997), once they have been taken into account (Gorard 1998), and assuming a census or high-quality sample is used, in a sense it does not matter where the study in located in England and Wales. It would therefore be a sign of progress if studies from Wales were no longer faced with the question 'does this only apply to Wales?' in situations where studies from London or Lancashire are not faced with an equivalent query.

If one accepts as broadly true the notion that quantitative analysis is good at establishing what the differences actually are in the attainment of boys and girls, and that qualitative analysis is better at explaining how and why these differences occur, then qualitative research in this area has a problem derived from its quantitative analytical framework. The majority of British work has been attempting to explain a pattern that does not, in fact, exist. This finding may have implications for the conduct of educational research in general, as well as specific points to make about the apparent under-achievement of boys.

General research literature attempting to explain the determinants of the gaps is relatively plentiful, but recent work is overwhelmingly concerned with a pattern of boys' 'underachievement' which is conceptualised as a general phenomenon, applying to boys as a relatively homogeneous category (or perhaps slanted towards lower levels of attainment). Proposals have included the need for more male teachers at primary schools, the value fo seating boys anf girls alternately, the use of after-school crammers for boys, and so on. For a review of this literature see Salisbury et al. (1999). Given the nature of previous research findings in this field, it is not surprising that proposals with respect to ameliorative action have focused on raising the level of boys' performance generally. Some more focused proposals have also been advocated. It has been has recommended, for example, that particular attention be paid to issues such as: identifying and supporting underachieving individuals at KS3 and KS4; improving classroom management with the specific objective of integrating boys and girls more effectively; setting targets both for individual students and subject departments; monitoring the effects of setting arrangements; and so on. A number of action-research projects have adopted strategies of mentoring to improve the performance of boys. Similarly, considerable attention has been directed at strategies for enhancing boys' language competencies.

The evidence is simply not available as yet to allow judgements to be formed on the effectiveness of these kinds of measures. It might be difficult to argue against them in general terms, but it is important to emphasise that they are not directed - and presumably are not intended to be directed - at addressing the complex pattern of differential attainment which the Cardiff study has revealed. The conclusions of much of the relevant work are in the form of suggested explanations. Some have very little empirical basis, many are based on very small-scale work and nearly all are directed at explaining the prevalent picture of gender gaps in school performance. Since that picture is now in dount, then even the most convincing explanations of that picture are in doubt, and their implications for remedial action inappropriate.

There are currently numerous initiatives aimed at ameliorating the incidence of achievement gaps through various approaches to mentoring, the targeting of 'under-achievers', the enhancement of boys' language competencies, single-sex setting and so on. It is clearly important to monitor the results of these programmes. However, in many cases, it seems unlikely that adequate data will be made available to allow proper evaluation. Even so, the desperate need for such studies may be glimpsed in the way in which a study of two teachers in one school (Woolford and McDougall 1998) was built by the media into a panacea requiring immediate changes in policy (Western Mail 5/2/98 p.1, TES 6/1/98 p.21). There is a case, therefore, for the implementation of properly constituted action-research projects designed to investigate the true impacts of these kinds of initiative. At the most basic level, there is a clear need to monitor the differential performance of boys and girls in assessment over time. In some areas, such as GNVQ, this will entail the generation of more detailed information than is currently available. The current confusion in this field is helping to build a moral panic about the underachievement of boys. Left unchecked, this panic might (and in some cases already does) influence the allocation of government and local authority finance, or even the funding of educational research.

Arnot, M., David, M. and Weiner, G. (1996) Educational Reform and Gender Equality in Schools, Manchester, Equal Opportunities Commission

Bentley, T. (1998) Learning beyond the classroom, London: Routledge

Bright, M. (1998) The trouble with boys, The Observer, 4/1/98, p. 13

Budge, D. (1999) Gulf separating weak and strong increases, Times Educational Supplement, 30/4/99, p. 3

Carvel, J. (1998) Help for boys lagging behind girls at school, The Guardian, 7/1/98, p.19

Clark, J., Modgil, C. and Modgil, S. (1990) John Goldthorpe: consensus and controversy, London: Falmer

Darroch, J. (1974) Multiplicative and additive interactions in contingency tables, Biometrika, 61, 2, 207-214

Erikson, R. and Goldthorpe, J. (1991) The constant flux: a study of class mobility in industrial societies, Oxford: Clarendon Press

Everitt, B. and Smith, A. (1979) Interactions in contingency tables: a brief discussion of alternative definitions, Psychological Medicine, 9, 581-583

Gibson, A. and Asthana, S. (1999) Schools, markets and equity: access to secondary education in England and Wales, presentation at AERA Annual Conference, Montreal

Gilbert, N. (1981) Modelling society: an introduction to loglinear analysis for social researchers, London: George Allen and Unwin

Gillborn, D. and Gipps, C. (1996) Recent research on the achievements of ethnic minority pupils, London: OFSTED

Goldthorpe, J., Llewellyn, C. and Payne, C. (1987) Social mobility and class structure in modern Britain, Oxford: Clarendon Press

Gorard, S. (1998) Four errors.... and a conspiracy? The effectiveness of schools in Wales, Oxford Review of Education, 24, 4

Gorard, S. (1999a) Keeping a sense of proportion: the "politician's error" in analysing school outcomes, British Journal of Educational Studies , 47, 3

Gorard, S. (1999b) For England see Wales: the distinctiveness and similarities of education in England and Wales, in Phillips, D. (Ed.) The education systems of the United Kingdom, Oxford: Oxford University Press

Gorard, S. and Fitz, J. (1998) The more things change.... the missing impact of marketisation, British Journal of Sociology of Education, 19, 3, 365-376

Gorard, S. and Fitz, J. (1999) Investigating the determinants of segregation between schools, Research Papers in Education , 14, 4

Gorard, S., Salisbury, J., Rees, G. and Fitz, J. (1999a) The comparative performance of boys and girls at school in Wales, Cardiff: Qualifications, Curriculum and Assessment Authority for Wales

Gorard, S., Salisbury, J. and Rees, G. (1999b) Reappraising the apparent underachievement of boys at school, Gender and Education , 11, 3

Gorard, S., Rees, G. and Fevre, R. (1999c) Families and their participation in learning over time, British Educational Research Journal , 25, 4

Gorard, S., Rees, G. and Fevre, R. (1999d) Two dimensions of time: the changing social context of lifelong learning, Studies in the Education of Adults , 31, 1, 35-48

Independent (1998) Classroom rescue for Britain's lost boys, The Independent, 5/1/98, p. 8

Lauder, H., Hughes, D., Watson, S., Waslander, S., Thrupp, M., Strathdee, R., Simiyu, I., Dupuis, A., McGlinn, J. and Hamlin, J. (1999) Trading in futures: Why markets in education don't work, Buckingham: Open University Press

Levacic, R., Hardman, J. and Woods, P. (1998) Competition as a spur to improvement? Differential improvement in GCSE examination results, presented to International Congress for School Effectiveness and Improvement, Manchester

Lieberson, S. (1981) An asymmetrical approach to segregation, in Peach, C., Robinson, V. and Smith, S. (Eds.), Ethnic segregation in cities, London: Croom Helm

Marshall, G., Swift, A. and Roberts, S. (1997) Against the odds? Social class and social justice in industrial societies, Oxford: Clarendon Press

Rees, G., Fevre, R., Furlong, J. and Gorard, S. (1997) History, place and the learning society: Towards a sociology of lifetime learning, Journal of Education Policy, 12, 6, 485-497

Robinson, P. and Oppenheim, C. (1998) Social Exclusion Indicators, London: Institute of Public Policy Research

Salisbury, J. , Rees, G. and Gorard, S. (1999) Accounting for the differential attainment of boys and girls: a state of the art review, School Leadership and Management , 19, 4

Slater, J., Dean, C. and Brown, R. (1999) Learning gulf divides rich and poor pupils, Times Educational Supplement, 2/4/99, p.1

Woolford, H. and McDougall, S. (1998) The teacher as role model, presentation to British Psychological Society Conference, Department of Psychology, Swansea University (mimeo)

 

This document was added to the Education-line database 16 September 1999