Education-line Home Page
BERA Conference, 1996

Handling Data: An Introductory Paper

Keith Morrison

There are three themes that this workshop will explore, through three projects.

1. Some issues in addressing reliability and validity;

2. How to address the social situatedness of data, their collection and analysis - the voiced and unvoiced dimensions of situations that yield data for researchers;

3. How to catch the complexity of situations and contexts in which data, researchers and respondents are located.

Though the projects are very different from each other, nevertheless they provide different examples of similar issues. That there is a tension between reliability and validity is common knowledge in the field of assessment - the more one has of the former the less one has of the latter and vice versa. In research terms also there is a delicate task of balancing the need to capture the richness and detail of the unique situation - the validity of the research - with the need to enable one to have confidence in the data and the findings from them - the reliability of the data, together with making the amount of the research data manageable. This workshop addresses the issues of reliability and validity in the three projects reported here.

It is a truism that data are socially situated because they derive from social situations. For example the interview is a social situation as well as a data-gathering instrument; it is an event as well as a source of data. People act on the basis of perceptions that they have of situations (recalling Thomas's much quoted aphorism from The Child in America that 'if men [sic] define their situations as real they are real in their consequences'). People bring the baggage of their own experiences, meanings, opinions, perceptions and biographies to situations. People act on the basis of the constructs that they hold about a situation. Meanings co-exist with phenomena. This is, of course, a cornerstone of interactionist research that builds in intentionality, agency, negotiation and the dynamics and specificity of situations, even though, by so doing, it might risk relativism.

There is a real risk in handling research data that the contents of the data become the sole focus of the research to the neglect of the way in which content might be mediated by the context of the data collection enterprise. As Habermas (1984) argues, there is a 'double hermeneutic' whereby, as researchers, we are dealing with an already interpreted world (pp. 109-110). Handling data, then, involves taking account of how the data were generated - the context of the data. If we are to understand a social situation with a degree of confidence this suggests that we use several conceptual, methodological and analytical lenses. This, in turn, suggests that the social situation as well as the contents of the data should figure in the data collection and analysis. It also suggests the need for multiple iterations of data to catch multiple interpretations of what is taking place.

Our first example - of naturalistic research - addresses the three themes clearly. Initially a feedback interview / discussion with a secondary school student teacher and university tutor were analysed. However, recognising that differential power relations were part of the situation, that the negotiation of a working consensus involved both excluding as well as including of foci and agendas, that silence might be as significant as spoken comments, suggested that a second round of data collection might be necessary in order to capture further details to understand the initial situation. Further, because the initial interview / discussion was conducted with a participant observer, for the second round an uninvolved researcher conducted subsequent interviews with two of the parties; they were asked to review the post-teaching interview / discussion, as an attempt to explain and understand the data further, giving reliability as well as authenticity to the data.

Our second example is of the gendered construction of special educational needs. Summarising the research, an exploration was conducted into the view that more boys than girls were classed as having special educational needs. Stage one of the research was using interviews to generate teachers' constructions of Special Educational Needs. In stage two - the focus of this workshop - questionnaires and pro-formas given to teachers were completed, that described the special needs of primary school children, what the learning difficulties were and who had learning difficulties. However it became clear that teachers' constructs of special educational needs were gendered. More boys than girls were seen as having special needs and teachers gave a multiplicity of reasons for classifying children with special needs, most of which were not specific to learning but more concerned with non-cognitive, background factors in children. This rehearses the view that cognitive psychology might provide us with an account of successful learning but social psychology might provide us with an account of learning failure. The third stage of data collection was conducted through interviews that attempted to gain further insight into why respondents had given their particular responses in the questionnaires and pro-formas. In order to gain some insight into the complexity of the issues and to attend to the reliability of the findings - the degree of confidence that could be placed in them, the researcher recognised the need for a multi-stage process of data collection and analysis. To do justice to the embeddedness of gender constructions of special educational needs in teachers' own perceptions and understandings required a wide data source to catch the context of those constructions. The understanding of special educational needs is the outcome of an understanding of the social situation of teachers and children.

Our third example - of 'external' research - is rooted in the practicality of secondary school religious education and reports young persons' attitudes to, and understanding of, exactly what happens (and happened) at Christmas, including the religious, commercial, and material aspects of Christmas. It is a form of action research. To address the issue of reliability a wide data source and multiple methodologies were used. A quantitative survey questionnaire to 9,500 teenagers was followed up with large scale qualitative data from over 300 teenagers writing freely about Christmas - the focus of this workshop. The focus of the issues was derived from teaching experience, lesson observations and discussions with students and teachers. However, the search for rich and authentic data exacts its price - the problem of handling large scale qualitative data without becoming too positivistic - a criticism levelled at Huberman's and Miles's much cited work Qualitative Data Analysis. We shall be asking you for your views about how these qualitative data might be processed and analysed.

So here we have three very disparate projects, yet each, in their way, offers some comments on three major themes in handling data:

1. How to address reliability and validity;

2. How to address the social situatedness of data, their collection and analysis - the voiced and unvoiced dimensions of situations that yield data for researchers;

3. How to catch the complexity of situations in which data and respondents are located so that we can understand them and maximise the explanatory potential of the data.

It seems as though one of the problems in handling data in order to address both reliability and validity in qualitative data - to do justice to the complexity of the situation and the phenomena being researched - is the size of the data set, for each of the examples suggest, in their separate ways, the need for a substantial, multi-staged project using a range of instruments. How this problem might be addressed is something about which we will welcome discussion. Contexts - observable features and mental constructs and perceptions of researchers, participants and audiences - exert important influences on the handling of data; the significance of these influences will be explored in the workshop.

The format of the presentations will be similar. Each project will be described and issues raised in handling data will be explored. Suggestions will be sought for how each project and data analysis could proceed - what the next stage of the project and data analysis might be.

Huberman, M. and Miles, M. (1984) Qualitative Data Analysis. Beverley Hills, Sage.

Habermas, J. (1984) The Theory of Communicative Action, Volume One: Reason and the Rationalisation of Society. London: Heinemann.

Thomas, W. I. (1928) The Child in America. New York, Knopf.

EMERGING ISSUES IN HANDLING DATA

1. Reliability and manageability might be in tension with each other - reliable findings might suggest the need for many data sources, with the problem of data overload.

2. Authenticity / validity render reliability as replicability and generalisability problematical.

3. Researchers' and participant's responses to the task and the data are influenced by their own values, knowledge and awareness of audience.

4. The data might be influenced by the participants' perceptions of the power / status of the researcher.

5. Framing a question (research and actual questions) and task might frame the response and outcome. This exerts an influence on the data handling; how far should the handling of the data be aware of the context(s) - observable and mental - and methods of data collection? We need to ask 'what do we need to know about the contexts of the data and their collection in order to understand the data?'.

6. To what extent should we take pieces of data in isolation and/or in the context of the other data?

7. There is a need for multiple iterations of data to enable multiple interpretations, categories and priorities to emerge / be identified. Data can be understood in many, sometimes contradictory, ways.

8. Even small pieces of data are complex. Data are seldom 'clean'.

9. Handling data is a messy and an inherently inexact task, with trade-offs between manageability and complexity. We usually need more data than we have.

10. Transcripts usually lose a variety of different types of data.