Effects of schools and classes upon mathematics and language achievement: the importance of group composition
Jan Van Damme, Bieke De Fraine, Marie-Christine Opdenakker, Georges Van Landeghem and Patrick Onghena
Secondary and Higher Education Research Centre
tel. +32 (0)16/32 62 45
fax. +32 (0)16/32 62 74
Paper presented at the European Conference on Educational Research, Edinburgh, 20-23 September 2000
This school effectiveness study focuses on differences in math achievement and language achievement at the end of the second grade of secondary education. A multilevel design with three levels (an individual level, a class level and a school level) was used. The results indicate that the group composition at the school level and especially at the class level is very important.
2 Research questions, data and method
We will start with some information on the system of secondary education in Flanders, the Dutch speaking part of Belgium, and on the research questions, the data and the method used.(1)
2.1 Secondary education in Flanders
The Flemish secondary educational structure contains six grades. Each pair of two grades is called a cycle. The first cycle is rather comprehensive. Only during a limited number of hours a week, optional subjects can be chosen. The majority of the students follows the general track, called the A-stream. Students with learning difficulties are advised to follow the B-stream. They receive education at pace with their ability. From the third grade on, four different categories of study programmes are distinguished, namely academic or general (GSO), technical (TSO), artistic (ASO) and vocational secondary education (VSO).
2.2 Research questions
Do schools make a difference? This question relates to the domain of school effectiveness, a quick expanding research domain. It started in the USA (Edmonds, 1979, Brookover et al., 1979) and the UK (Rutter et al., 1979). But more and more European countries (The Netherlands, Belgium, ...) gather data and study school differences within the educational context of their country.
In this school effectiveness study, the following research questions were addressed. Firstly, to what extent is the school relevant in explaining the differences in achievement for mathematics and for language? In other words: do schools matter?
Secondly, to what extent are classes and teachers relevant in explaining the differences in mathematics and language achievement? Most studies just include two levels: schools and students. In their meta-analysis of 168 studies, Bosker and Witziers (1996) address the magnitude of school effects, ignoring the possible effect of classes and teachers. Opdenakker and Van Damme (2000a) make clear that ignoring a level can lead to wrong conclusions. Some say that the impact of the school on the students is mostly indirect, trough teaching and learning processes in classes. Teachers and classes largely account for school effects (Scheerens & Creemers, 1989).
Thirdly, which part of the variation at the school and class level is due to recruitment, i.e. to selective entrance in schools and classes? Flemish schools differ a lot in their recruitment. Private (most catholic) schools tend to have students with a more favourable socio-economic background than public schools. And more important: there are still schools with only general (academic) or only technical and vocational education. To make a fair estimation of the variability at the school and the class level, relevant predictors at the individual level are included in the model.
Fourthly, which characteristics of schools and classes are relevant in explaining the net effects of schools and classes? At both levels two types of characteristics are considered. The first category of explanatory variables contains variables referring to the composition of the group. The second category of explanatory variables refers to the educational processes within the class or school, especially an indicator of the learning climate.
In answering the final research question the explanatory value of both types of variables will be compared (contrasted). Recently, the question was raised as if variance in climate goes back to variance in group composition (Thrupp, 1999).
2.3 Data: The LOSO project
From 1990 on, Van Damme et al. (1997) studied a cohort of more than 6000 students in 57 Flemish secondary schools. This Longitudinal Research in Secondary Education Project (in Dutch, LOSO) was funded by the Department of Education of the Ministry of the Flemish Community. The general purposes of this longitudinal study were to describe and explain the interindividual differences within school career paths, in the perspective of the evaluation of the entire Flemish educational system on the one hand and of the identification of the major determinants of school effectiveness on the other hand.
The students were followed since September 1990 for a -in the first phase- period of seven years. The sample consists of all students who entered secondary education for the first time in the first grade of these schools. The schools are to a certain extent representative of the schools in Flanders in general. The programmes offered and the distribution of the students over the programmes is comparable to the situation in Flanders as a whole.
Not only the students were questioned. Their teachers, their parents and the headmasters of the schools they attended were other important sources of information. Data were gathered at the individual level (intelligence, motivation, SES, achievement in mathematics and Dutch, ...), at the teacher level (teaching style, differentiation, ...), at the class level (composition, size, ...) and at the school level (students', teachers' and headmaster's report of school life, student body composition, ...). (For more information on the data, see the next parts of our text and see also Opdenakker & Van Damme, 2000b).
For the analyses, a subset of 4759 students was selected. These students followed the second grade of the A-stream in 1991-1992, grouped in 275 classes in 57 secondary schools. There are 147 math teachers and 155 Dutch teachers, so some teachers teached more than one class. Therefore the data have in fact a four level structure: individuals within classes within teachers within schools. The two intermediate levels are combined by a random selection of one class per teacher. After this selection was made, we distinguish three levels in the data: the student, the class and the school level. Important is that the class level combines the teacher and the classgroup.
The analysis of math achievement will be performed using a subsample of 2324 students in 147 classes, teached by 147 teachers in 56 schools. The selected dataset for the analysis of Dutch achievement consists of 2569 students in 152 classes in 55 schools.
2.4.2 Multilevel analysis
Because of this grouping structure, a multilevel design is necessary (Bryk & Raudenbush, 1992; Kreft & de Leeuw, 1998). In multilevel models, the variance is partioned between the different levels at which it is measured. To analyse the data, the MLwiN-software (Rasbash, J. et al., 2000) was used.
Through multilevel analysis, the impact of several variables at the three levels was tested. The multilevel analysis was done stepwise. First a null model, without explanatory variables, is fitted to provide estimates of the variation at each of the three levels. This null model shows how the total variance in mathematics or Dutch achievement is partitioned into three variance components.
In the next steps we gradually included explanatory variables at the different levels. We started at the individual level. Seven student characteristics were included in the model: initial cognitive ability, SES of the family, achievement motivation, fear of failure, sex, language spoken at home, and achievement (math/Dutch) at the end of first grade. The inclusion of relevant variables at the student level is necessary to correct for differences in school and classroom recruitment. As already mentioned, Flemish schools differ a lot in their recruitment: some schools offer only academic programmes and others offer only technical and vocational programmes. Private (mostly catholic) schools tend to have more students with a more favourable socio-economic background than public schools. By adding student level variables to the multilevel model, we avoid an overestimation of the importance of the schools and classes due to their selective entrance. This inclusion makes an estimation of the net effects of schools and classes possible.
Once this net effect had been estimated, variables of the higher levels are added to the model, first the class and teacher characteristics and secondly the school characteristics, to explain the class and school related variability. This stepwise multilevel analysis distinguishes the relevant from the irrelevant predictors. Finally, a model with only the relevant variables was tested. In this final model we examined the possible existence of random slopes among schools and among classes within schools. Also the hypothesis of heteroscedasticity was tested within this final model. There is heteroscedasticity when a variance component is not constant, but a function of some of the explanatory variables.
All explanatory variables were centered around the grand mean of their corresponding level. This makes the computer calculations easier and the parameter estimations are more comprehensible than those stemming from the raw scores model (Opdenakker & Van Damme, 1997).
2.5.1 Dependent variables : achievement in mathematics and Dutch
This paper refers to two dependent variables: scores on a mathematics test and scores on a language test (Dutch). These school achievement tests were administered at the end of second grade (A-stream). These are curriculum specific multiple choice tests which were constructed for this project. Both tests were approved by a board of inspectors and teachers. The mathematics test covers theory of numbers (26 items) and geometry (18 items). The internal consistency is 0.70. The Dutch test covers grammar (17 items), linguistic performance (16 items), reading comprehension (29 items) and spelling (22 items). The internal consistency is high (0.90).
The raw scores on both tests were converted into IRT-scores, situating both students and test scores on two latent scales, for mathematics and Dutch respectively.
2.5.2 Explanatory variables at the individual level
Seven explanatory variables at the individual level were selected: initial cognitive ability, SES, achievement motivation, fear of failure, sex, language spoken at home and achievement in mathematics or Dutch at the end of first grade. This selection was based upon a combination of educational theory and prior empirical evidence. Most of these variables were constructed using higher order principal component analysis. The component 'initial cognitive ability' can be seen as a combination of scores on an intelligence test an scores on a mathematics test, both administered at the start of first grade. The socio-economic status of the family includes the educational level and the occupation of both father and mother. Families are also characterised by the language at home: in some families only the Dutch language is spoken, in others also other languages are used. This variable is a raw indicator of the immigrant status of the family.
2.5.3 Explanatory variables at the class/teacher level
Some of the above mentioned variables at the individual level were used in an aggregated manner to describe the group composition. By calculating the mean of the student characteristics for each class separately, we constructed several group composition variables at the class level: the average initial cognitive ability, the average prior math achievement, the average achievement motivation, the average SES and the proportion of girls in the class. Another descriptor of the class group was the option chosen.(2)
Process characteristics were based on information by the teachers of the subjects math and Dutch. On the one hand we used the description of the class group by the teacher on different aspects such as the extent in which the students in the class group were a calm, a study-oriented and a cohesive group. Through principal component analysis these different evaluation aspects were combined in one higher order factor. This factor was called a steady learning climate.
On the other hand the teachers were asked to give a description of their teaching practice. The following variables were explored: focus on individual development (7 items,α=0.63 for math and α=0.67 for Dutch teachers), to structure explicitly the subject matter (4 items, α=0.76, only for math teachers), special attention to weakly (1 item) and strongly (1 item) achieving students, the inclusion of more deep level questions in tests (1 item, only for math teachers), consultation between teachers (higher order combination of consultation on students (3 items, α=0.76 for math and α=0.76 for Dutch teachers) and consultation on didactics (4 items, α==0.82 for math and α=0.84 for Dutch teachers), and feedback on study results (individual feedback, 1 item, feedback in group, 1 item).
An additional variable is the "opportunity to learn" and it indicates the degree to which items of the achievement test at the end of the second grade were covered during the school year.
2.5.4 Explanatory variables at the school level
The school type refers to the study programmes offered by the school. Multilateral schools offer with general, technical and vocational secondary education. These multilateral schools will be compared with schools that offer only general or only technical and vocational education. Also private (catholic) and public schools will be compared.
A similar technique of calculating means of student characteristics, as described for the class level, was used to describe the student population of a school. The averages of initial cognitive ability, SES and achievement motivation were derived. The proportion of girls within the school is another group composition characteristic that will be included in the multilevel design.
Fifteen process variables at the school level were selected for inclusion. For the construction of these variables information from different sources was combined (students, teachers and headmaster). On the one hand we used the answers of the students and the teachers on a school characteristics questionnaire and the description of several aspects of the school environment by the headmasters of the schools. These answers and descriptions were analysed using first and higher order principal component analysis. The following school variables were constructed using the answers of the teachers (a representative sample of 15 teachers of each school) on the questionnaire: attention towards differences between students, focus on discipline and subject matter acquisition, orderly learning environment and teaching staff co-operation, the use of test results to improve teaching and an evaluation of the functioning of the school in general. The latter variable was also measured using the data of the school characteristics questionnaire for students. Using the data of the questionnaire for the headmaster we constructed a variable referring to the attention to pedagogical aspects and student counselling and a variable referring to the extent to which formal structures and regulations are typical for the school.
On the other hand variables at school level were constructed by aggregating some of the variables at class level: average level of inclusion of deep level questions in tests, average level of structure in subject matter, average level of special attention to weakly and strongly achieving students, average level of consultation between teachers, average level of feedback on study results and the average level of opportunity to learn.
We will now present the two multilevel models that were estimated. Firstly, the mathematics achievement at the end of second grade will be discussed and in the following paragraph the multilevel model of the Dutch achievement will be presented.
3 The effect of schools and classes upon mathematics achievement
The null model shows that schools and classes within schools differ a lot with respect to the mathematics achievement of their students. The schools account for 29% of the total variance in mathematics achievement and 26% of the variance is situated at the class/teacher level.
To make a fair comparison, based on the value added, we first added students' initial cognitive ability. Having taken this initial ability into consideration, there are still significant contributions of SES, achievement motivation and math achievement at the end of the first grade. We found no differences between boys and girls and between students from Dutch- and non-dutchspeaking families. The net school differences were 13.2%.
As for the student characteristics, they allow to explain about 40% of the total variance in math achievement at the end of the second grade (see Figure 1). Twenty-two percent is explained by the students' initial cognitive ability.
Figure 1. Variance in math achievement explained by significant student characteristics
Figure 2 shows that these student characteristics explain only a small part of the variance within each class, but they do explain a major part of the variance between classes within schools and between the schools. Schools and classes indeed recruit different student bodies.
Figure 2. Variance in math achievement explained by significant student characteristics by level
Our research indicates that explaining mathematics achievement of second grade students is a complex enterprise. Based on correlational and multilevel analyses we conclude that many important class and school characteristics correlate with each other and have net and joint effects. The composition of the class group (in terms of SES and intelligence), which correlates with indicators of good teaching practice and classroom climate, has a remarkable positive effect on mathematics achievement. On the other hand, students benefit from a class climate that is calm, focused on learning and cohesive, having a teacher with positive expectations towards the achievement level of the students. This learning climate is highly correlated with the composition of the group. The common factor of group composition and class climate explains 5% of the total variance and the pure group composition is accountable for 6% of the variance, whilst the climate in se explains only 1% (see Figure 3).
Figure 3. Overlap between learning climate and aggregated student characteristics
The relevant class characteristics allow to explain most of the variance between classes which was not related to student characteristics. They also allow to explain half of the remaining variance at the school level (see Figure 4).
The rest of the differences between schools can be explained by some school characteristics. Firstly we saw that the higher the percentage of girls in school, the more effective the school. Secondly, in schools that organise both the academic and the technical/vocational track (multilateral schools), the achievement is higher than in schools with only academic or only technical/vocational sections. Thirdly, the more attention the teachers pay to differences between students, the less the performance of these students. This strange outcome shows that we are dealing with correlations and that causal interpretations are difficult. The right interpretation is probably that in schools with low math scores, teachers tend to differentiate more between their students. We found no relevant influence of the schools' denomination, discipline and emphasis on cognitive achievement, and composition of the school population (in terms of SES, intelligence and achievement motivation).
When all the relevant variables at the three levels are included in the model, almost none of the enormous variation between Flemish schools is left. This means we can nearly explain all the differences between schools by student characteristics (63%), class and teacher variables (19%) and school variables (15%).
Figure 4. Variance in math achievement explained by significant student, class and school characteristics, by level
Our results about the effect of group composition and class climate confirm to a certain extent the points of view of Martin Thrupp (1999), who stresses the impact of between school social class segregation.
Also some interesting interactions were found; a.o. an interaction between sex and a positive learning climate in the classroom.
Figure 5. Student level variance as a function of sex and learning climate
As shown in Figure 5, only girls seem to be sensitive to climate characteristics, as the achievements of girls are rather homogeneous in a class with a positive learning climate and their achievements are rather heterogeneous in a class with a negative learning climate.
4 The effect of schools and classes upon language achievement
In this paragraph, we explore the effects of the school, the class/teacher and the individual student upon language achievement in Flanders.
The null model shows that 31% of the variance in Dutch achievement is accounted for by schools, 24% lies at class/teacher level and 44% at the student level. This indicates major differences between schools and between classes/teachers within schools.
Flemish schools differ a lot in their recruitment. To make a fair comparison we introduce student SES, initial cognitive ability, sex and achievement motivation. We also correct for the Dutch achievement at the end of the first grade of secondary school.
Figure 6. Variance in Dutch achievement explained by significant student characteristics
These student characteristics explain about 55% of the achievement at the end of the second grade (see Figure 6). The student characteristics are thus more important for language than for mathematics achievement. In contrast with mathematics, we found major differences between boys and girls, with a better language achievement for the girls. Having taken these student characteristics into consideration, we did not find a relationschip between the language at home and the Dutch achievement of the student.
As can be seen in Figure 7, the relevant student characteristics explain more than 70% of the variance at class and school level. The background variables explain a larger proportion of between-school variance and between-classroom variance than of pupil level variance. This points to the homogeneity of schools and of classes within schools. Recruitment accounts for a great deal of the differences between classes and between schools.
Schools and classes are more or less equally important for Dutch achievement, even when the effects are corrected for differences in recruitment.
Figure 7. Variance in Dutch achievement explained by significant student characteristics
At the teacher and class level, two group composition variables, average initial cognitive ability and proportion of girls, are of major influence. The variables relating to the student body composition have a greater impact than the process characteristics. These two class composition variables explain some of the variance at class/teacher level and 10% of the school variance, because schools are homogeneous with respect to the percentage girls in their classes. The net school effects are partially due to the influence of class composition.
At the school level there is no extra influence of the school composition variables and no relevant influence of school process variables. The schools' denomination makes a minor difference: in public schools, students tend to obtain higher grades (when correcting for the relevant student and class variables).
When all the relevant variables at the three levels are included in the model, almost none of the enormous variation between schools is left, as can be seen in Table 1. We can nearly explain all the differences between schools by student, class, teacher and school variables.
Table 1. Variance in Dutch achievement at the three levels
variance explained by student variables
variance explained by class variables
variance explained by school variables
Further specification of the model indicates that the girls are a more homogeneous group than the boys with respect to their Dutch achievement.
Second grade classes differ in the strenght of the relationship between Dutch achievement at the end of first grade and the end of second grade.
Figure 8. The relationship between Dutch achievement at the end of first grade and Dutch achievement at the end of second grade, per second grade class
In some classes (the more steep lines in Figure 8), the results at the end of first grade are largely reproduced at the end of second grade. In other classes (the more horizontal lines in Figure 8), the Dutch achievement at the end of first grade is not a very good predictor for the Dutch achievement, one year later. Especially for low achievers, the class and the teacher are important.
Some important conclusions of our study are that, as for the achievement level,
1. the class and teacher level is at least as important as the school level;
2. group composition and process characteristics (esp. climate characteristics) are linked but there are indications that group composition characteristics are more important than process characteristics;
3. when not only the individual differences between the students but also the group composition effects are taken into account, the achievement level in public schools is not worse than that in private schools.
Bosker, R.J., & Witziers, B. (1996). The magnitude of school effects. Does it really matter which school a student attends? Paper presented at the annual meeting of the AERA. New York.
Brookover, W.B., Beady, C., Flood, P., Schweitzer, J., & Wisenbaker, J. (1979). School social systems and student achievement. Schools can make a difference. New York: Praeger.
Bryk, A.S., & Raudenbush, S.W. (1992). Hierarchical linear models. Applications and data analysis methods. Newbury Park: Sage.
Edmonds, R. (1979). A discussion of the literature and issues related to effective schooling. Cambridge, MA: Center for Urban Studies, Harvard Graduate School of Education.
Kreft, I. & de Leeuw, J. (1998). Introducing Multilevel Modeling. London: Sage
Minnaert, A. (1991). WIPREESO Schoolvorderingentoets wiskunde eind eerste leerjaar A secundair onderwijs (Mathematic achievement test end of first year A secondary education). Leuven: K.U.Leuven, Afdeling Psychodiagnostiek en Psychologische Begeleiding & Onderzoekscentrum voor secundair en hoger onderwijs.
Opdenakker, M.-C., & Van Damme, J. (1997). Centreren in multilevelanalyse: implicaties van twee centreringsmethoden voor het bestuderen van schooleffectiviteit (Centering in multilevel analysis: implications of two centering methods for the study of school effectiveness). Tijdschrijft voor Onderwijsresearch, 22(4), 264-290.
Opdenakker, M. C., & Van Damme, J. (2000a). The importance of identifying levels in multilevel analysis: an illustration of the effects of ignoring the top or intermediate levels in school effectiveness research. School Effectiveness and School Improvement, 11(1), 103-130.
Opdenakker, M. C., & Van Damme, J. (2000b). Effects of Schools, Teaching staff and Classes on Achievement and Well-being in Secondary Education in Belgium: Similarities and Differences between School Outcomes. School Effectiveness and School Improvement, 11(2), 165-196.
Rasbash, J., Browne, W., Goldstein, H., Yang, M., Plewis, I., Healy, M., Woodhouse, G., Draper, D. & Lewis, T. (2000). A user's guide to MLwiN. University of London, Institute of Education, Multilevel Models Project.
Rutter, M., Maughan, B., Mortimore, P., & Ouston, J. (1979). Fifteen thousand hours: secondary schools and their effects on children. Cambridge, M.A.: Harvard University Press.
Scheerens, J., & Creemers, B.P.M. (1989). Conceptualizing School Effectiveness. In Developments in school effectiveness research (special issue). International Journal of Educational Research, 13, 691-706.
Thrupp, M. (1999). Schools making a difference. Let's be realistic. Buckingham: Open University Press.
Van Damme, J., De Troy, A., Meyer, J., Minnaert, A., Lorent, G., Opdenakker, M.-C., & Verduyckt, P. (1997). Succesvol doorstromen in de aanvangsjaren van het secundair onderwijs (Successful passing through the first years in secondary education). Leuven: ACCO.
1. This research was sponsored by the Department of Education of the ministry of the Flemish Community.
2. In the first cycle their is a high focus on basic and common education. Only during a limited number of hours subjects can be chosen by the student.