|
|
ABSTRACT
The theory of fuzzy logic suggests a way of encapsulating the claims to educational knowledge of qualitative empirical research. A fuzzy generalisation replaces the certainty of a scientific generalisation (‘it is true that ...’) by the uncertainty, or fuzziness, of statements that contain qualifiers (‘it is sometimes true that ...’). [It contrasts with the statistical generalisation of quantitative empirical research (‘it is true in p% of cases that ...’)]
Alone a fuzzy generalisation is no more than the researcher’s equivalent of the politician’s sound-bite, and as such has little credence, but, supported by a research account which makes clear the context of the statement and the justifying evidence, it provides a user-friendly account of research findings.
Fuzzy generalisation invites replication and this, by leading to augmentation and modification of the generalisation, contributes powerfully to the edifice of educational theory.
PERSONAL BACKGROUND
In 1980 I gave a paper at the BERA conference on the problems of generalising in educational research and at about the same time a related paper to CARN with the title ‘Crocodiles eat Children’. This was a reference to the example of a generalisation used by Ausubel and Robinson (1971) - in a widely used undergraduate text on educational psychology - which I pilloried, saying:
In my view Ausubel and Robinson’s example of a generalisation, in their textbook School learning, is peculiarly apt. I suspect that every general statement made about school learning has the same property of lack of certainty. (Bassey 1980)
A year later the Oxford Review of Education published an extended version of my BERA paper, in which I challenged some of the general statements arising from empirical research conducted by leading educational researchers of the time, and drew attention to:
the paucity of generalisations which are relevant to classrooms (Bassey 1981:77)
I discussed the value of studies of single events (which in a later paper, following Simon, 1980, I called ‘studies of singularities’) and suggested that the relatability of a research study (ie the extent to which a reader could relate it to their own practice) was a more useful concept than generalisability.
The Oxford Review paper was reprinted in an Open University reader. (Bell et al 1984)
Early this year I re-examined my previous thinking in the course of preparing a book on case study (Bassey, in press) and changed my view of generalisation. This paper draws on the manuscript of the book and develops further some of the ideas contained in it.
CONCEPTS OF GENERALISATION IN THE CASE STUDY LITERATURE (1)
Louis Cohen and Lawrence Manion first published Research Methods in Education in 1980. It has been reprinted many times and is now into its fourth edition: it is probably the widest used source book on research methods used by education students in the UK. The authors devoted 30 of some 400 pages to a chapter on Case Studies. Their stance was this:
Unlike the experimenter who manipulates variables to determine their causal significance or the surveyor who asks standardised questions of large, representative samples of individuals, the case study researcher typically observes the characteristics of an individual unit - a child, a clique, a class, a school or a community. The purpose of such observation is to probe deeply and to analyse intensively the multifarious phenomena that constitute the life cycle of the unit with a view to establishing generalisations about the wider population to which that unit belongs. (1989:124-125)
However not everybody sees generalisation as an essential outcome. Lawrence Stenhouse, for example, writing the article on Case Study Methods in the first edition of Keeves (ed, 1988) Educational Research, Methodology and Measurement: an International Handbook, was more concerned about producing case reports on which the reader could exercise judgement. He said:
Case study methods involve the collection and recording of data about a case or cases, and the preparation of a report or a presentation of the case. ... Sometimes, particularly in evaluation research, which is commissioned to evaluate a specific case, the case itself is regarded as of sufficient interest to merit investigation. However case study does not preclude an interest in generalisation, and many researchers seek theories that will penetrate the varying conditions of action, or applications founded on the comparison of case with case. Generalisation and application are matters of judgement rather than calculation, and the task of case study is to produce ordered reports of experience which invite judgement and offer evidence to which judgement can appeal. (p49)
A decade later, in the second edition of Keeves handbook (1997) the article on Case Study Methods was written by A Sturman, an Australian researcher with considerable experience of writing case studies. He focused attention on the holistic nature of cases and the need for the study of them to investigate the relationships between their component parts. He said:
‘Case study’ is a generic term for the investigation of an individual, group or phenomenon. While the techniques used in the investigation may be varied, and may include both qualitative and quantitative approaches, the distinguishing feature of case study is the belief that human systems develop a characteristic wholeness or integrity and are not simply a loose collection of traits. As a consequence of this belief, case study researchers hold that to understand a case, to explain why things happen as they do, and to generalise or predict from a single example requires an in-depth investigation of the interdependencies of parts and of the patterns that emerge. (p61)
The familiar criticism facing case study researchers is ‘How can you generalise when n=1?’.
Robert Yin in the United States is probably the leading exponent in the social sciences of case study. He first published Case Study Research: Design and Methods in 1984; there were 13 additional printings and a second edition in 1994. In terms of generalising in order to create theory his book refers to ‘statistical generalisation’ (which is unsuitable for case studies) and ‘analytic generalisation’ (which can be appropriate). Of the former he gave a firm warning:
In statistical generalisation, an inference is made about a population (or universe) on the basis of empirical data collected about a sample ... a fatal flaw in doing case studies is to conceive of statistical generalisation as the method of generalising the results of the case. (1994:30-31).
This would seem to be exactly what Cohen and Manion were advocating!
Yin argued that analytic generalisation is the appropriate method for generating theory from case study, by which he meant that
a previously developed theory is used as a template with which to compare the empirical results of the case study. If two or more cases are shown to support the same theory, replication may be claimed. (1994:31)
In the late 1970s Adelman, Jenkins and Kemmis recognised that ‘generalisation’ might be an equivocal term and suggested that in relation to case study there are three kinds of generalisation:
The first kind is from the instance studied to the class it purports to represent (eg a study of comprehensivisation in one school may tell us about comprehensivisation in other schools). The second kind is from case-bound features of the instance to a multiplicity of classes (eg a study of comprehensivisation in one school may tell us about leadership or relations with the press in other schools: this is my example MB]. Studies which do not begin by asserting the instance-class relation, however, will be inclined towards the third kind of generalisation: generalisations about the case. (In Simons 1980:50)
Stenhouse in his presidential address to the British Educational Research Association in 1979, with typical farsightedness, described the co-existence of two ‘cloven heads’ in educational research. He said that he would ‘try asserting’ that:
the most important distinction in educational research at this moment is that between the study of samples and the study of cases. (Stenhouse 1980:2)(2)
He added:
I believe that the description of cases and the analytic categorisation of samples are complementary and necessary approaches in educational research, and it is high time that the superficial stylistic differences between their proponents were recognised as impediments to good sense in the research community. (1980: 4)
He expected both to lead to generalisation. He distinguished between predictive generalisation and retrospective generalisation. Predictive generalisation is that which arises from the study of samples and is the form in which data are accumulated in the sciences: it is what Yin called ‘statistical generalisation’. Retrospective generalisation is that which can arise from the analysis of case studies and is the form in which data are accumulated in history: in Yin’s terms this is ‘analytic generalisation’. Stenhouse was concerned about teacher’s classroom judgements and earlier had noted that:
while predictive generalisations claim to supersede the need for individual judgement, retrospective generalisations seek to strengthen individual judgement where it cannot be superseded. (Stenhouse 1978)
Stake (1995) in his book The Art of Case Study Research. expressed worries about generalisation:
Case study seems a poor basis for generalisation ... The real business of case study is particularisation (pp7-8)
and then invented a new range of terminology about it.
He suggested the term petite generalisation for general statements made within a study - for example that a particular child responds repeatedly in the same way to a particular situation. And he recognised that grand generalisations, meaning general statements about issues of which the case is one example, can be modified by the findings of the particular case.
Instead of making grand generalisations, Stake argued for researchers drawing from their research conclusions in the form of assertions (which later he called ‘propositional generalisations’).
Interpretation is a major part of all research. ... the function of the qualitative researcher during data gathering is clearly to maintain vigorous interpretation. On the basis of observations and other data, researchers draw their own conclusions. Erickson called them assertions, a form of generalisation.’ (Stake 1995:9)
These assertions, he noted, will often be petite generalisations (ie located within the case study) but occasionally may refer to wider populations and so be grand generalisations (p20).
He went on to discuss how research data are interpreted in the form of assertions and reflected on how researchers may fail to make clear the speculative and tentative nature of their assertions.
We do not have adequate guides for transforming observations into assertions - yet people regularly do it. ... The logical path to assertions often is apparent neither to reader nor to researchers themselves. ... For assertions, we draw from understandings deep within us, understandings whose derivation may be some hidden mix of personal experience, scholarship, assertions of other researchers. It will be helpful to the reader when such leaps to conclusions are labelled as speculation or theory, but researchers often do not.’ (p9, p12)
Stake was gentle when he castigated fellow researchers for overstating their findings, but the importance of this paragraph is ignored at our peril:
It is not uncommon for case study researcher to make assertions on a relatively small database, invoking the privilege and responsibility of interpretation. To draw so much attention to interpretation may be a mistake, suggesting that case study work hastens to conclusions. Good case study is patient, reflective, willing to see another view of the case. An ethic of caution is not contradictory to an ethic of interpretation. (p12)
What beautiful language!
Stake introduced in 1982 (with Trumbull) the term ‘naturalistic generalisation’. By this they meant ‘conclusions arrived at through personal engagement in life’s affairs’ (1995, p86). Here they were using the term ‘generalisation’ to refer to the learning processes through which we individually acquire concepts and information and steadily generalise them to other situations as we learn more. Stake noted that Hamilton called this ‘an inside-the-head generalisation’ (p86). He also noted that naturalistic generalisation can be made through ‘vicarious experience [if it is] so well constructed that the person feels as if it happened to themselves’ (p86). It is the provision of this vicarious experience that he sees as a key role for case study writers.
To assist the reader in making naturalistic generalisations, case researchers need to provide opportunity for vicarious experience. Our accounts need to be personal, describing the things of our sensory experiences, not failing to attend to the matters that personal curiosity dictates. A narrative account, a story, a chronological presentation, personalistic description, emphasis on time and place provide rich ingredients for vicarious experience. (p87)
Stake contrasted these naturalistic generalisations, which are made personally by the reader, with the ‘propositional generalisations’ (or assertions) made publicly by the researcher . He urged case researchers to consider with care how much of their writing should provide input for the reader’s naturalistic generalisations and how much should spell out the researcher’s propositional generalisations. He recognised, of course, that the reader will do both, taking narrative descriptions to provide vicarious experience leading to naturalistic generalisations, and taking propositional generalisations (or assertions made by the researcher) alongside existing propositional knowledge to modify of extend it.
In 1985 an Australian researcher with considerable experience of classroom research, David Tripp, published a paper entitled ‘Case Study Generalisation: an agenda for action’. His concern was with how research findings can be applied to classrooms by practising teachers and he argued for a cumulative process in bringing case studies together. He saw this as ‘qualitative generalisation’ (what Stake called ‘naturalistic generalisation’) in which the individual, meeting the facts of a new case, applies them to their knowledge of similar cases, and so develops personal understanding. In order to do this Tripp argued that it is important for each case report to document carefully the salient features of the case. He suggested that there are two kinds of salient features: ‘comparable’ features (eg every classroom case study giving sex, age, ability and socio-economic status of the pupils, an account of the teaching facilities in the classroom and the teaching style of the teacher) and ‘comprehensive’ features (particular circumstances which are judged relevant to the events observed, etc). He then said:
If several thousand case studies were in an archive, then the problem to emerge is how the studies could be made available so that a researched case appropriate to the needs of an inquirer may be found to illuminate an unresearched situation. (p37)
Supposing that that were resolved by precise indexing of salient features and findings, Tripp then envisaged
teachers with a particular problem first going to the archive to find how it occurs and has been dealt with elsewhere, then acting upon that vicarious experience in their own classrooms before finally documenting their experience to be added to the archive. (p41)
Finally I would like to refer to a recent paper by Helen Simons entitled ‘The Paradox of Case Study’ (1996) in which she welcomes the paradox between the study of the singularity and the search for generalisation.
One of the advantages cited for case study research is its uniqueness, its capacity for understanding complexity in particular contexts. A corresponding disadvantage often cited is the difficulty of generalising from a single case. Such an observation assumes a polarity and stems from a particular view of research. Looked at differently, from within a holistic perspective and direct perception, there is no disjunction. What we have is a paradox, which if acknowledged and explored in depth, yields both unique and universal understanding. (p225)[We need to] embrace the paradoxes inherent in the people, events and sites we study and explore rather than try to resolve the tensions embedded in them. ... Paradox for me is the point of case study. Living with paradox is crucial to understanding. The tension between the study of the unique and the need to generalise is necessary to reveal both the unique and the universal and the unity of that understanding. To live with ambiguity, to challenge certainty, to creatively encounter, is to arrive, eventually, at ‘seeing’ anew. (pp237-238)
In developing her argument she draws effectively on the use of paradox in painting and poetry and ends with lines from T S Eliot’s Four Quartets:
We shall not cease from explorationAnd the end of all our exploring
Will be to arrive where we started
And know the place for the first time.
Eighteen years after first speaking at BERA on generalisation I feel now that I am arriving where I started - and do know more about it than I did. I wish I had not been so dogmatic in taking the scientific definition of ‘generalisation’, which led me to miss the opportunity of building in ‘lack of certainty’ to the definition.
The above canter through some of the literature only looks at generalisation through the eyes of the case study researcher and I recognise that there are many other writers whose work has a bearing on this issue whom I have ignored. I hope to address their concerns in a later paper.
This was the background that lead me to the following conceptualisation of research.
THE CONCEPTS OF FUZZY GENERALISATION AND STATISTICAL GENERALISATION
By ‘scientific generalisation’ I mean the kind of absolute general statement that has to be rejected if one contrary instance is found that challenges its verity. In its simplest form it is like this: if x happens y follows. This is the Popperian view of scientific laws (Popper 1963) and good science is that which tries to refute, rather than support, such statements. In my attacks on generalisation of 18 years ago this is the kind of generalisation that I was challenging as far as education is concerned. I am still sure that I was right that there are few such statements that tell teachers anything worthwhile.
But there are other forms of generalisation. A statistical generalisation is one which says something like this: in p% of cases it will be found that x leads to y. It arises as the result of a careful sampling of a population and it can be expected that any other careful sample of the same population will give the same result.
I have recently proposed that the term ‘fuzzy generalisation’ should be used for the kind of statement that says: in cases similar to the cases studied it may be found that x leads to y. (Bassey 1998) [Below I elaborate on the choice of ‘fuzzy’]. This may arise from one case study - or better several. There is unmeasured uncertainty in the description.
Consider situations in which certain actions do, or do not, lead to particular results: the terminology to be used is:
s = situation sr = randomly chosen situation so = opportunistically chosen situationx = action
y = particular result of action
Starting from a fuzzy proposition
Suppose that in a chosen situation (so1) the carrying out of x leads to y.
The fuzzy proposition is drawn that in other situations like so it is possible that x may lead to y.
There are two approaches to testing this further in order to ascertain how likely it is that x generally will lead to y: the search for a statistical generalisation or the search for a fuzzy generalisation.
Search for a statistical generalisation
Here the academic emphasis is on defining the characteristics of x and y, identifying the population of situations where x may occur, and then randomly sampling this population when x is carried out.
Suppose that in ten randomly chosen situations (sr1 to sr10) the results of carrying out x are as follows:
in sr1 x leads to yin sr2 x leads to y
in sr3 x doesn’t lead to y
in sr4 x leads to y
in sr5 x leads to y
in sr6 x leads to y
in sr7 x leads to y
in sr8 x leads to y
in sr9 x leads to y
in sr10 x doesn’t lead to y
The statistical generalisation is drawn that in situations like sr there is an 80% chance that x will lead to y.
Search for a fuzzy generalisation
Here the academic emphasis is also on defining the characteristics of x and y, but in each case in terms of a detailed analysis of the situation and of the question ‘why does x lead to y?’
Suppose that a couple of replications are carried out in chosen situations so2 and so3 and it is found that:
in so2 x leads to yin so3 x leads to y
The fuzzy generalisation is drawn that in other similar situations x is likely to lead to y.
There is no statistical measure of ‘is likely to’.
Modification of a fuzzy generalisation through replication study
Suppose that in a further replication at so4 it is found that:
in so4 x does not lead to y
This is where an important leap forward in understanding may be made. The researchers examine in detail not only what happened in so4, but go back through so1, so2 and so3 and try to modify the description of x to find an x’ such that:
in so1, so2, so3 and so4 ‘x’leads to y
FUZZINESS
I was struggling to find a way of expressing succinctly the idea of a qualified generalisation when I came across a paper by C Fourali called ‘Using fuzzy logic in educational measurement’. (Fourali 1997) This resolved for me a problem with which I have often grappled as an examiner of student papers. Instead of trying to give an exact mark - like 57 out of 100 for an essay, Fourali advocated giving a fuzzy mark, like 50-60 out of 100. If another examiner gave a fuzzy mark of 55-70, then it might be appropriate to combine the two and give a narrower range of 55-60 as the moderated mark. Then it dawned on me that this was what I was looking for: my ‘qualified’ generalisation could be described as a ‘fuzzy’ generalisation.
A popular text on fuzzy logic is that of Bart Kosko - ‘Fuzzy Thinking’ (1994). [He attributes the word ‘fuzzy’ to Lofti Zadeh, who began publishing on fuzzy sets in the 1960s, and chose the word in preference to ‘vague’] Kosko links the word ‘fuzzy’ to principles, sets, logic, systems, the past, the future, and much else! ‘The fuzzy principle states that everything is a matter of degree.’ (p18) He doesn’t use the term but I can see nothing in his writing that would quarrel with the concept of fuzzy generalisation as I have used it here.
RELATIONSHIP BETWEEN EMPIRICAL FINDING AND GENERALISED CONCLUSION
I like to separate the empirical finding of a paper from any generalised conclusion that comes from it. Two examples may be helpful.
Empirical finding leading to a statistical generalisation
Peter Blatchford and Clare Sumpner recently published a paper (Blatchford and Sumpner 1998) based on a questionnaire survey in 1996 which included 1245 primary schools in England. One of their findings was that in answer to the question: ‘Has pupil behaviour changed out of school between 1990/91 and 1995/96?’ 55% of their sample responded ‘declined’, 35% said ‘not changed’ and 5% said ‘improved’. This is an empirical finding.
Because there was a careful sampling of schools this could lead to a statistical generalisation in this form: the opinion of primary school teachers in England in 1996 on change in pupil behaviour out of school over the past five years was that 55% considered it to have declined, 35% considered it not changed, and 5% considered it improved. [It is proper to add that Blatchford and Sumpner did not make such a generalisation from their empirical finding - no doubt feeling that the latter spoke for itself] Where this statistical generalisation would become important is if in a few years time a replication study (with a different random sampling of the population) was conducted and differences between ‘then and now’ were drawn.
Empirical finding leading to a fuzzy generalisation
Helen Morgan recently published a report (Morgan 1997) of in-depth interviews of some sixth-form students carried out in Denbigh School in 1996. She found that these sixth-form students, through talking and answering questions about their learning experiences, were able to give help to their teachers towards making improvements in their teaching. This is an empirical finding.
This could lead to the following fuzzy generalisation: sixth-form students, through talking and answering questions about their learning experiences, may give help to their teachers towards making improvements in their teaching. [Again it is proper to note that Morgan did not make such a generalisation - this is my formulation of the outcome of her work] Suppose that other teacher-researchers carry out replications of this study in their own schools. Some might find difficulties that Morgan did not experience. It could be, for example, that a modification to the above fuzzy generalisation gets introduced by adding ‘... provided that the school has a tradition of open discussion between staff and students.’
FITTING THESE IDEAS INTO AN OVERALL PICTURE OF EMPIRICAL EDUCATIONAL RESEARCH
The chart on the next page sets the above ideas in the context of a wider concept of empirical educational research (Bassey, in press). The white boxes contain the ideas developed in this paper. It is suggested that (following Stenhouse 1980:2) there are two arenas for educational research: studies of singularities and studies of samples. Case studies, experiments and action researches are examples of the former, surveys of the latter. It is also suggested that there are two kinds of outcome: predictions and interpretations. Fuzzy generalisations and statistical generalisations constitute the former, stories (narrative/analytical accounts) and pictures (descriptive/analytical accounts) constitute the latter.
REFERENCES
Adelman C, Kemmis S, Jenkins D (1985) Rethinking Case Study: notes from the second Cambridge conference, in Simon (1980) (ed) 45-61, loc cit
Bassey M (1980) Crocodiles eat children, Bulletin of Classroom Action Research Network
Bassey M (1981) Pedagogic research: on the relative merits of search for generalisation and study of single events, Oxford Review of Education, 7(1) 73-94 [Reprinted in Bell et al (1984) loc cit]
Bassey M (1998) Enhancing teaching through research, Professional Development Today, 1, 2, 39-46 (A shortened version was published in Research Intelligence, July 1997)
Bassey M (in press) Case Study Research In Educational Settings, Buckingham, Open University Press
Bell J, Bush T, Fox A, Goodey J and Goulding S (eds) (1984) Conducting small-scale investigations in educational management London, Harper and Row
Blatchford P and Sumpner C (1998) What do we know about breaktime? Results from a national survey of breaktime and lunchtime in primary and secondary schools British Educational Research Journal 24, 1, 79-94
Cohen L and Manion L (1989) Research Methods in Education (3rd ed) London, Routledge
Fourali C (1997) Using fuzzy logic in educational measurement, Evaluation and Research in Education 11, 3, 129-148
Keeves J P (1985) (ed) Educational Research, Methodology, and Measurement: an international handbook, 1st edition, Oxford: Pergamon
Keeves J P (1994) (ed) Educational Research, Methodology, and Measurement: an international handbook, 2nd edition, Oxford: Pergamon
Kosko B (1994) Fuzzy Thinking, London, Harper/Collins
Morgan H (1997) Motivation of Sixth-Form Students, London, TTA
Popper K (1963) Conjectures and Refutations, Oxford University Press
Simons H (1980) (ed) Towards a Science of the Singular, Centre for Applied Research in Education, University of East Anglia, Norwich
Simons H (1996) The Paradox of Case Study, Cambridge Journal of Education, 26, 2, 225-240
Stake R E (1995) The Art of Case Study Research, London: Sage
Stenhouse L (1978) Case Study and Case Records: towards a contemporary history of education, British Educational Research Journal 4(2) 21-39
Stenhouse L (1980) The Study of Samples and the Study of Cases, British Educational Research Journal 6(1) 1-6
Stenhouse L (1985) Case Study Methods, in Keeves (1985) (ed) 49-53, loc cit
Sturman A (1994) Case Study Methods, in Keeves (1994) (ed) 61-66, loc cit
Tripp D (1985) Case Study Generalisation: an agenda for action, British Educational Research Journal 11,1, 33-43
Yin R K (1994) Case Study Research: Design and Methods, (2nd edition) London: Sage
Correspondence: The Cottage, The Moor, Kirklington, Newark, NG22 8NQ email 101464.1532@compuserve.com
1. Bold typeface in this section is not that of the quoted authors, but mine.
2. I use this idea later, but interpete it as the study of samples and the study of singularities
This document was added to the Education-line database 28 September 1998