Some Questions about Evidence-based Practice in Education
Faculty of Education and Language Studies, The Open University, Walton Hall, Milton Keynes MK6 7AA
Paper presented at the symposium on "Evidence-based practice in education" at the Annual Conference of the British Educational Research Association, University of Leeds, England, September 13-15, 2001
The movement for evidence-based practice, for enhanced use of research evidence in the work of the professions, started in medicine in the early 1990s. It has grown in influence there, and spread across a number of other fields, including education (see Trinder 2000a and Davies et al 2000). Not surprisingly, it has taken a somewhat different form in each area; and has generated diverse reactions. So, while much can be learned from what has happened in medicine, we must also attend to the particular way evidence-based practice has recently been promoted and responded to in education, if we are to make a reasonable appraisal of it.
There is an initial, and generic, problem with the notion of evidence-based practice which needs to be dealt with. This is that its name is a slogan whose rhetorical effect is to discredit opposition. After all, who would argue that practice should not be based on evidence (Shahar 1997:110)? So there is an implication built into the phrase 'evidence-based practice' that opposition to it can only be irrational. Interestingly, critics managed to counter this by denying that practice can be based on evidence; in the sense that research evidence can provide its exclusive foundation. The response to this on the part of advocates has been to change "evidence-based" to "evidence-informed" practice(1). At face value, this suggests a more reasonable view of the relationship between research and practice. However, it is at odds with the radical role that is ascribed to research by the evidence-based practice movement. As a result, even more than before, we have a label that systematically obscures the grounds on which there might be reasonable disagreement with what is proposed.
In political terms, as a way of mobilising support, the use of such opposition-excluding labels is no doubt a highly effective rhetorical strategy. However, it is a poor basis for rational discussion about the issues that the notion of evidence-based practice raises. Against this background, it is very important to emphasise that one can believe that research evidence is of value for practice without accepting much of what travels under the heading of 'evidence-based' or 'evidence-informed' practice; indeed, while rejecting substantial parts of it. So, let me say to start with that I do think that, on the whole and in the long run, practice would be improved if practitioners were more familiar with the results of research. I also accept that there is scope for more directly policy- and practice-relevant educational research. Furthermore, I applaud the emphasis within some accounts of evidence-based practice on the need for professional reflection and judgement about the validity and value of various kinds of evidence in making decisions(2). At the same time, there are problems and dangers with the evidence-based practice movement. These result from tendencies built into it, tendencies that have an affinity with other pressures in the current political environment. One of these dangers is the privileging of research evidence over evidence from other sources, especially professional experience.
The privileging of research evidence
The central claim of the evidence-based policy movement is that research can make a very important contribution to improving the current state of policymaking and practice. Thus, 'evidence' is interpreted in a rather narrow way. In their introduction to What Works? Evidence-based policy and practice in the public services, Davies et al comment: 'the presumption in this book is that evidence takes the form of "research", broadly defined. That is, evidence comprises the results of "systematic investigation towards increasing the sum of knowledge"' (Davies et al 2000:3). In some fields, notably medicine, the movement has been primarily (though even here not exclusively) concerned with encouraging practitioners to make more use of the research evidence that is already available. Initiatives like the Cochrane Collaboration, concerned with producing and disseminating systematic research reviews, are seen as facilitating this; and the training of medical practitioners has, in many institutions, come to place more emphasis than previously on the skills involved in accessing and making use of research findings. By contrast, in other fields, but most notably education, greater emphasis has been placed on the absence of good quality research which could feed evidence-based practice (see, for example, Hargreaves 1996).
The idea that research can make a major contribution to improving practice stems from the assumption that it is systematic and rigorous, and provides explicit evidence which can be assessed objectively. This is held to contrast with evidence from professional experience, which is portrayed as unsystematic - reflecting the particular cases with which a practitioner has happened to come into contact - and as lacking in rigour - in that it is not built up in an explicit, methodical way but rather through an at least partially unreflective process of sedimentation. Indeed, sometimes the contrast is presented in a form that can only be described as caricature, as with Cox's reference to teachers often relying on 'tradition, prejudice, dogma, and ideology' (quoted in Hargreaves 1996:7-8)(3). Such caricature complements the already mentioned rhetorical sleight of hand built into the very name of the evidence-based practice movement.
Moreover, this view of the role of research fits with a progressivist philosophy in which evidence-based practice is seen as opposing the "forces of conservatism" in the public sector, forces which are taken to represent entrenched interests. For example, Oakley claims that the medical profession, along with the pharmaceutical industry, have 'a vested interest in women's ill-health - in defining women as sick when they may not be, and in prescribing medical remedies when they may not be needed' (Oakley 2000:51). These interests are seen as protected by the claim of professionals to a kind of expertise which cannot be communicated or shared with lay people but which instead demands professional autonomy and public trust(4). In this paper I will argue that we need to look much more soberly both at the features of research-based knowledge compared to those of knowledge deriving from practical experience; and at how research findings relate to professional practice.
The nature of research-based knowledge
Let me begin with what can be currently said about the character of research-based knowledge and its relationship to practice(5). It is important to remember that research knowledge is always fallible, even if it is more likely to be valid than knowledge from other sources. Thus, we are not faced with a contrast between Knowledge with a capital K, whose epistemic status is certain, and mere opinion, whose validity is zero(6). Furthermore, research knowledge usually takes the form of generalisations, of one sort or another, and interpreting the implications of these for dealing with particular situations is rarely straightforward. A final point is that knowledge is not a sufficient determinant of good practice, in education or in any other field. One reason for this is that it cannot determine what the ends of good practice should be; or even, on its own, what are and are not appropriate means. Furthermore, the effectiveness of any practical action usually depends not just on what is done but also on how it is done and when. Skill and timing can be important. In other words, there are substantial limitations on what research can offer to policymaking and practice. This is not to suggest that it can offer nothing, but rather to caution against excessive claims about its contribution(7).
The nature of professional practice
It seems to me that built into some advocacy of evidence-based practice is not just an exaggerated estimate of the practical contribution that research can provide but also a misleading conception of the nature of professional practice. Very often, it is assumed that the latter should take the form of specifying goals explicitly, selecting strategies for achieving them on the basis of objective evidence about their effectiveness, and then measuring outcomes in order to assess their degree of success (thereby providing the knowledge required for improving future performance).
This model is not wholly inaccurate, but it is defective in important respects. Forms of practice will vary in the degree to which they can usefully be made to approximate this linear, rational model(8). And it probably does not fit any sort of professional activity closely. Reasons for this include: that such activity usually involves multiple goals; that these goals cannot be operationalised properly in a way that avoids calling on professional judgement; that the same action has multiple consequences, some desirable and others less so, these being differentially distributed across clients; and that there is frequently uncertainty surrounding the likely consequences of many strategies, even in a field like medicine where a great deal of clinical research is carried out. As result of these features, there can often be reasonable disagreement about what would be an improvement and about what sorts of improvement are to be preferred, as well as about how these can best be achieved. Moreover, sometimes it will simply not be sensible to engage in elaborate explication of goals, to consider all possible alternatives, to engage in a search for information about the relative effectiveness of various strategies as against relying on judgements about this, or to try to measure outcomes. The linear rational model tends to underplay the extent to which in many circumstances the only option is trial and error, or even 'muddling through' (Lindblom 1979).
In this context, we should note that the very phrase 'what works', which the evidence-based practice movement sees as the proper focus for research, implies a view of practice as technical: as open to "objective" assessment in terms of what is and is not effective, or what is more and what is less effective. I would not want to deny that effectiveness, and even efficiency, are relevant considerations in professional practice. But the information necessary to judge them in the "objective" way proposed will often not be available. And any such judgement cannot be separated from value judgements about desirable ends and appropriate means; not without missing a great deal that is important.
Furthermore, there is a significant difference between medicine and education in terms of the nature of professional practice, it seems to me. For whatever reason, much medicine is closer to the technical end of the spectrum, in the sense that there is less diversity in the goals and other considerations treated as relevant; and thereby in evaluative criteria. Furthermore, there seems to be more scope for identifying relatively simple causal relationships between treatment and outcome. Of course, it is possible to exaggerate these differences. In response to my presentation of this argument on a previous occasion, Davies has claimed that 'medicine and health care [...] face very similar, if not identical, problems of complexity, context-specificity, measurement, and causation' to education (Davies 1999:112). I do not deny that there are such problems in medicine; and that in some areas, for example mental health, they are very similar in scale and character to those faced in much education. What we are dealing with here is only a difference of degree; but it is still a substantial and significant difference.
In short, in my view research usually cannot supply what the notion of evidence-based practice demands of it - specific and highly reliable answers to questions about what 'works' and what does not - and professional practice often cannot be governed by research findings - because it necessarily relies on multiple values, tacit judgement, local knowledge, and skill. Moreover, this is especially true in the field of education. When pressed, advocates of evidence-based practice often concede one or other, or both, of these points. Yet those points undermine the claim that improved research, or a reformed version of professional practice which gives more attention to research findings, will lead to a sharp improvement in educational performance and outcomes.
The usability of research findings
It is also important to address the question of how research findings are to be used by policymakers and practitioners. Let me mention two problems here. First, there is the issue of the transmission of research findings to these audiences. Some accounts of evidence-based practice imply that policymakers and practitioners are expected to read all relevant research studies, and to engage in methodological assessment of them, before deciding what to do in each case they face. Thus, we have various books produced to aid medical practitioners in assessing research (see, for example, Crombie 1996 and Greenhalgh 1997). However, as a general model for accessing research information, this is simply unrealistic: medical practitioners are unlikely to have the necessary time and, often, the appropriate skills; and the same is true of teachers.
The alternative suggested by many advocates of evidence-based practice is the production of systematic reviews of research relevant to decisions faced by policymakers and practitioners (see Davies 2000a). And this proposal certainly has the advantage that reliance is not placed on single studies, which often produce erroneous conclusions. However, reviewing multiple studies and condensing these into a set of practical conclusions is by no means a straightforward matter; nor is using such summaries.
The concept of systematic review shares some common elements with the notion of evidence-based practice. It portrays the task of reviewing the literature as reducible to explicit procedures that can be replicated; just as advocates of evidence-based practice see professional work as properly governed by explicit rules based on research evidence. Moreover, here too questionable assumptions are made about the nature of research, and about the task involved; ones which have long been subjected to criticism as 'positivist'. It is not necessary to doubt the value of quantitative research, or even to believe that positivism was wholly mistaken, to recognise that these criticisms have some force and cannot be simply ignored. Yet the concept of systematic review does not take account of them. For example, it approaches the task of assessing the validity of research findings as if this could be done properly simply by applying explicit and standard criteria relating to research design. Yet, validity assessment cannot rely entirely on information about research design. Much depends on the nature of the knowledge claims made, and assessment of them always relies on substantive knowledge as well as on specifically methodological considerations (see Hammersley 1998). The result of this mistaken approach to validity assessment is that systematic reviews are likely to exclude or downplay some kinds of study that may be illuminating, notably qualitative work, while giving weight to other studies whose findings are open to serious question (see Hammersley 2001).
An illustration of the attitude underlying the notion of systematic review, and the evidence-based practice movement generally, is Oakley's recent adoption of what can only be described as a form of naive positivism(9). This is surprising because she has been identified in the past as advocating qualitative methods, largely on feminist grounds. However, she is now a strong advocate of randomised controlled trials. She insists that she has never advocated qualitative method exclusively, and that she does not advocate quantitative method exclusively now. Nevertheless, in some parts of her recent book, Experiments in Knowing, she evidences a dismissal of qualitative work that is the obverse of her over-enthusiastic support for quasi-experimental research design. For example, she claims that the debate over Derek Freeman's critique of Margaret Mead's anthropological study of Samoa is best read as an example of the 'untrustworthiness' of 'uncontrolled findings' (Oakley 2000:57). The implication of this seems to be that all "uncontrolled" findings are untrustworthy. Later she comments: 'Recent attitudes to evaluation in some quarters have been marked by a retreat into more "naturalistic" and theory-driven approaches to public policy evaluation, but these serve the interests of academic careers more than they promote the goals of either an emancipatory social science or an informed evidence base for social and policy interventions' (Oakley 2000:323). Here we have dismissal of the views of those who disagree with her via the attribution of ulterior motives, and in the form of assertion rather than appeal to evidence. At one point she claims that anything else but randomised, controlled trials 'is a "disservice to vulnerable people" [...]' (Oakley 2000:318; quoting Macdonald 1996:21)(10). A more general problem about how research can be used by practitioners concerns the funnelling process through which research findings need to be refined into summary statements that policymakers and practitioners will have time to read and digest. This almost always involves the paring away of most qualifications and methodological information. While such summaries may be a very useful representational device for those who have already read the whole review, or the whole study, they may be obscure or misleading for those who have not done this. In some cases, the summary may be difficult to understand without background information. This is true of all kinds of research review. For example one of the key points of Gillborn and Gipps' review of the educational achievements of ethnic minority children was that '"colour-blind" policies have failed', but it seems clear that some audiences found this conclusion difficult to interpret (Gillborn and Gipps 1996:80)(11). Alternatively, summaries of research findings in reviews may seem easily interpretable, but the interpretation given may be some distance from that intended; and/or the validity of the findings may be dismissed at face value. For example, a study which suggests that there is no positive, linear correlation between amount of homework done and level of pupil achievement may be interpreted as saying that homework should be abolished or is not important(12). This stems from the fact that the interpretative framework within which policymakers and practitioners approach research findings is often rather different from that of researchers; in particular these audiences are likely to be much more directly concerned with practical implications. It is also possible that summary findings will be seen by policymakers or practitioners as trivial, in the sense of repeating what is already known, the actual contribution to current knowledge only being clear against the background of the study or review as a whole.
Another problem concerns how research findings are to be related to what policymakers and practitioners think they already know. This problem can again be illustrated by David Blunkett's response to research findings about homework. He dismissed these on the grounds that they ran counter to his beliefs. He was wrong to dismiss them, and to believe that research which produces such findings must be unsound. However, he would not have been wrong to have approached those findings (like any others, especially from a single study) with caution. And there is a genuine problem for practitioners about how to weigh contradictory evidence, especially when it has been produced in different ways. In his argument for evidence-based practice, Davies comments: 'There is no question of evidence replacing clinical judgement or experience, but of uniting these two dimensions of knowledge to provide a sound basis for action' (Davies 1999:111). Yet conflicting evidence from different sources cannot be 'united' in any simple sense(13). And evidence from professional experience cannot be assessed in the same way as research evidence; since the process by which it was produced is not documented. It is difficult to see what one can do, when faced with contradictory evidence from research and practical experience, other than either trust one's instincts, which is the Blunkett strategy, or assume that research evidence is always sounder than that from experience, which is what advocates of evidence-based practice sometimes seem to imply. Neither approach is satisfactory. What is perhaps required is that each source of evidence be subjected to internal scrutiny in its own terms: reflection on the sources, relations, and functions of particular beliefs, in the case of professional experience; and methodological assessment, in the case of research. However, this is a time-consuming process; and it by no means guarantees that the result will be a satisfactory resolution of any contradiction between the two. Moreover, any resolution will necessarily rely on judgement that cannot be fully explicated(14). Up to now, I have looked at the role of research in relation to policymaking and practice in rather abstract terms. However, it is important to locate the evidence-based practice movement, and reactions to it, in the current political context.
Evidence-based practice and 'transparency'
The evidence-based practice movement is closely related to influential demands for 'transparent' accountability that are characteristic of what has come to be called managerialism, or the 'new public management' (Ferlie, E., et al. 1996; Pollitt, 1990; Mayne and Zapico-Goni. 1997; Power, 1997). As Davies et al note: 'In contrast to the preceding culture of largely judgement-based professional practice, there has arisen the important notion of evidence-based practice as a means of ensuring that what is being done is worthwhile and that it is being done in the best possible way' (Davies et al 2000:2). Thus, it is assumed that research can "ensure" that the best is being done; both by providing information about "what works", and by documenting whether practitioners are actually following "best practice"(15). Moreover, research is believed to be capable of doing this because it is objective and explicit; what it provides is open to public scrutiny, in a way that professional experience and judgement are not.
The demand for 'transparent' accountability seems to have arisen from two sources, though my account here is rather speculative. The first source lies in the field of commerce and industry, and concerns the emergence of 'generic management'(16). Where, in the first half of the twentieth century, managers had often worked all of their careers in a single sector, frequently moving up to managerial positions within an individual firm, in the second half of the century there was increasing mobility of managers across sectors, and a growing tendency to define management in terms of generic rather than sector-specific knowledge and skills(17). Along with growth in the average size of firms (see Devine et al 1985:85-90), often as a result of mergers and take-overs, this led to the problem of how senior managers were to assess the performance of different parts of their organisations. The solution proposed was to find objective indicators of performance, in other words indicators that did not rely on direct observation of the work being done and whose interpretation did not demand detailed knowledge of what that work entailed. Over the course of the 1960s, 70s and 80s this reliance on performance indicators spread to the public sector (initially to the publicly owned utilities, for example in the energy field, later to health and education), on the back of arguments that this sector was inefficient by comparison with private enterprise(18). An equally important stimulus to the 'new public management', however, was recognition by the political Left of the force, and electoral appeal, of the argument that taxpayers had a right to know that their money was being well-spent. Public accountability of this kind now came to be seen as central to democracy, on the Left as well as on the Right. But, of course, most citizens were in exactly the same position as the generic manager, in that they did not have direct knowledge of what was involved in the work of public sector agencies. What they did have was some, differentiated, experience as clients of those organisations; and access to others' anecdotes about failures in service, which were elaborated by media campaigns against the 'inefficiency' of the public sector. Here, too, the solution proposed was objective performance indicators; indicators that would allow politicians and the general public to judge what was happening and how it could be improved. Indeed, the new public management portrayed government ministers as managers of the public sector, and therefore as responsible for monitoring performance and intervening to improve it. Furthermore, in the context of the New Labour government, politicians began to suggest that their own performance should be judged by whether promised improvements in the work of public sector organisations took place, as measured by performance indicators; an extension of transparent accountability encouraged by the media, for whom the results of such indicators constituted a useful source of news(19). More generally, criticism of the public sector for failing to 'deliver' a satisfactory level of service, and portrayal of it as inferior to the private sector, arose out of widespread attacks, in the second half of the twentieth century, on professionalism and on the role of the state. Once again, these came from both sides of the political spectrum, and from social scientists(20). The claim of professional occupations to be governed by a service ethic, which justified their having autonomy in performing their work, was challenged on the grounds that they had misused this autonomy to serve their own interests. There was some evidence of this; but the charge also reflected a change in the prevalent vocabulary of motives, whereby any claim to motives other than self-interest was subjected not just to suspicion but to disbelief. From the Right, in particular, it was insisted that people would only do what was in the general interest if circumstances were arranged in such a way that it was in their self-interest to do it. The market was seen as the model in this respect; a fact which fuelled not just privatisation but also the application of 'market discipline' within the public sector.
The assumption was that markets provide consumers with all the knowledge they need in order to assess the relative value of the goods on offer. As a result, their purchasing decisions reward efficiency and punish inefficiency. However, to a large extent, this is simply a myth. It does not even hold true where conditions of perfect competition are approximated, a situation which is very rare. This is because consumers do not usually have easy access to all of the information they would need to judge effectively the relative value of what is on offer. Reliable and comparable information about the relative quality of different products, even different products from the same supplier, is often not available(21). Moreover, most consumers would probably not have the background knowledge or the time necessary to make use of that information were it available. In fact, the tendency is for consumers to rely heavily on one kind of information: that which is most easily available and open to assessment - price. And this tendency distorts the production process, since lowering costs is given priority over ensuring quality(22).
Of course, the attempt to introduce transparent accountability in the public sector is not in any simple sense a process of marketisation. Instead, efforts are made directly to measure the quality of services; and not so much as a basis for consumer decision making (though this is sometimes the case, as with school league tables) but so as to allow the public to judge whether the public services are meeting satisfactory standards of performance and providing 'value for money'. However, for the reasons explained in the previous section, these performance indicators do not usually measure what is important effectively; and, indeed, advocates of transparent accountability will often acknowledge this, while insisting that they provide the best evidence available. Yet, ironically, the severe problems associated with such performance indicators were demonstrated long ago in the attempts of Eastern European governments to control centrally the production of goods and the provision of services. Here, even attempts to specify industrial production targets and measure performance for relatively simple goods failed (Nove 1980:ch4)(23).
Moreover, this parallel shows that efforts to render people accountable in terms of performance indicators distort the production process at least as much as does the emphasis on lowering cost and price which is characteristic of the private sector. This is because it encourages, indeed to a large extent forces, individual practitioners to adopt an instrumental orientation in which scoring highly on the indicators becomes more important than doing a good job in terms of their own judgement. Indeed, they may even lose the confidence to exercise such judgement.
Where the focus is professional services the problems are especially severe because of the level of specialised expertise and knowledge on which these services rely. Furthermore, what they involve is not a determinate process. For example, even the best surgeons do not succeed in all cases; and, to the degree that they deal with more difficult cases than their colleagues, their failure rates may be relatively high. In this context, it is not just a matter of judging a product or service in terms of whether it meets requirements, or meets requirements better than others - which is hard enough - but judging whether the best that could have been done has been done. In the field of education this task is further complicated by the fact that teachers deal with children in batches, not on a one-by-one basis. So, here, judgement has to be according to whether what was done was the best for all concerned, which may involve trading benefits for some against costs for others.
The application of transparent accountability to medicine, education, and other areas has been premissed on the assumption that explicit information can be provided about all the factors relevant to judging the quality of professional performance in these fields. And research is seen as playing a key role in this: the belief is that it can show what works and what does not work, thereby providing a standard against which the practice of professionals can be judged. Moreover, it is assumed that research is itself transparent, that it simply documents how things are in an objective fashion. However, for reasons already explained, research is unable to play this role. Furthermore, efforts to make it play this role seem likely to have undesirable consequences. In relation to practice, the appeal to research findings on all significant issues may undermine the value of practical experience and commonsense, and thus erode practitioners' confidence in their own judgements. While it is sometimes claimed that evidence-based practice represents an enhanced professionalism, its devaluing of professional experience and judgement relative to accountability in terms of externally produced research findings seems likely to lead to a weakening of professionalism, in most senses of that term. Equally important, the attempt to make research serve this function has the effects of privileging practical against academic or scientific research, and of eroding the distinction between research and the work of management consultancy, audit, and inspection agencies(24). There is no doubt that evaluations of professional practice can be valuable; or that some professional practice, in all fields, is poorer than it could be, for all manner of reasons. However, not only is the notion of transparent accountability a myth, but it relies on a version of perfectionism which implies that failures and risks can be avoided, or at least can be progressively reduced in number and seriousness. So, it is assumed that performance can and should always be further improved; and that measures can be taken to ensure that any failures that have occurred in the past do not recur in the future. In promising this, transparent accountability encourages a climate in which clients demand that their needs and wants be fully met, while practitioners are increasingly concerned simply with protecting themselves from possible criticism (not to mention possible legal action). And, given that the indicators do not measure performance effectively, especially not across the whole range of each profession"s activities, this is likely to worsen not to improve the quality of performance.
Advocates of evidence-based practice often deny that there is any link between it and attempts to introduce 'transparent accountability' in the public sector. However, this is naive at best. The Government sees each as serving the other, and is acting on that basis in seeking to bring about "reform". It views research as playing a major role in this. Hence the current attempt to bring educational research under increased central control, coordinating and concentrating funding on issues directly relevant to policy and practice (see NERF 2000; Hodkinson 2001).
In this paper I have argued that while few would disagree that professional practice could be improved if practitioners had better access to the products of a large body of relevant research, the evidence-based practice movement is not likely to lead to a dramatic improvement in educational performance. In fact, my argument suggests that it could even lead to a worsening in the quality of service. The reasons for this lie in misconceptions about the nature of both research and practice which are built into the assumptions on which it operates. Its advocates have too much confidence in the validity of research findings, both in abstract terms and in comparison with knowledge coming from professional experience. And they assume that research can play a much more direct role in relation to practice than it usually can. They tend to treat practice as the application of research-based knowledge; neglecting the extent to which it necessarily involves uncertain judgement. Moreover, there are some serious difficulties involved in the use of research evidence by practitioners. One relates to problems in interpreting this evidence without background knowledge about the studies from which it arose. Another concerns the question of how contradictions between research evidence and professional experience are to be resolved.
Finally, I pointed to the close contemporary association between the evidence-based practice movement and the new public management, with its efforts to make the public sector transparently accountable and thereby to improve its performance substantially. I argued that this notion of accountability is built on a mythology of the market that does not accurately capture how markets work, and that is peculiarly inappropriate to the case of providing services that necessarily rely on a good deal of specialised expertise. I suggested that the search for transparent accountability is a futile enterprise, and one which is likely to have negative consequences for both professional practice and research. It offers a false hope of dramatic improvement in quality, while at the same time undermining the conditions necessary for professionalism to flourish. The entanglement of the movement for evidence-based practice with these developments means that it should be treated with the greatest caution.
Aucoin, P. (1990) 'Administrative reform in public management: paradigms, principles, paradoxes and pendulums', Governance, 3, 2, pp115-37.
Blunkett, D. (1999) Secretary of State"s address to the annual conference of the Confederation of British Industry.
Blunkett, D. (2000) 'Influence or irrelevance: can social science improve government?', Research Intelligence, 71, pp12-21.
Burnham, J. (1941) The Managerial Revolution, Harmondsworth, Penguin.
Chandler, A. (1977) The Visible Hand, Cambridge, Harvard University Press.
Chandler, A., with Takashi Hikino, (1990) Scale and Scope: the dynamics of industrial capitalism, Cambridge MS, Harvard University Press.
Cochrane, A. L. (1972) Effectiveness and Efficiency, Nuffield Provincial Hospitals Trust.
Crombie, I. (1996) The Pocket Guide to Critical Appraisal, London, BMJ Publishing Group.
Davies, H. T. O., Nutley, S. M., Smith, P. C. (eds) (2000) What Works? Evidence-based policy and practice in the public services, Bristol, Policy Press.
Davies, P. (1999) 'What is evidence-based education?', British Journal of Educational Studies, 47, 2, pp108-121.
Davies, P. (2000a) "The relevance of systematic reviews to educational policy and practice", Oxford Review of Education, 26, 3/4, pp365-78.
Davies, P. (2000b) "Contributions from qualitative research", in Davies, H. T. O. et al (ed).
Devine, P. J., Lee, N., Jones, R. M., Tyson, W. J. (1985) An Introduction to Industrial Economics, London, Unwin Hyman.
Evans, J. (2001) 'Systematic reviews of educational research: Does the medical model fit?', British Educational Research Journal, forthcoming.
Ferlie, E., Ashburner, L., Fitzgerald, L. and Pettigrew, A. (1996) The New Public Management in Action, Oxford, Oxford University Press.
Gillborn, D. and Gipps, C. (1996) Recent Research on the Achievements of Ethnic Minority Children, London, Office for Standards in Education, HMSO.
Greenhalgh, T. (1997) How to read a paper: the basics of evidence-based medicine, London, BMJ Publishing Group.
Hammersley, M (1993) 'On the teacher as researcher', in M. Hammersley (ed) Educational Research: current issues, London, Paul Chapman.
Hammersley, M. (1997) "Educational research and teaching: a response to David Hargreaves" TTA lecture", British Educational Research Journal, 23, 2, pp141-61
Hammersley, M. (2000a) "The sky is never blue for modernisers: the threat posed by David Blunkett"s offer of “partnership” to social science", Research Intelligence, 72, June.
Hammersley, M. (2000b) 'Media representation of social and educational research: the case of a review of ethnic minority education', paper presented at the British Educational Research Association conference, Cardiff University, September.
Hammersley, M. (2002) 'On "systematic review" of research literature', unpublished paper.
Hammersley, M. (2002) Educational Research, Policymaking, and Practice, London, Paul Chapman.
Hampton, J. R. (1997) "Evidence-based medicine, practice variations and clinical freedom", Journal of Evaluation in Clinical Practice, 3, 2, pp123-31.
Hargreaves, D. (1996) Teaching as a reseach-based profession: possibilities and prospects, London, Teacher Training Agency.
Hodkinson, P. (2001) "Response to the National Strategy Consultation Paper, for the National Educational Research Forum", Research Intelligence, 74, February.
Lindblom, C. (1979) 'Still muddling, not yet through', Public Administration Review, 39, pp??.
Macdonald, G. (1996) "Ice therapy? Why we need randomised controls", in Barnados What Works? Effective social interventions in child welfare, Ilford, Essex, Barnados.
Marris, R. (1971) 'The economic theory of "managerial" capitalism', in Archibald, G. C. (ed) The Theory of the Firm, Harmondsworth, Penguin.
Mayne, J. and Zapico-Goni, E. (eds) (1997) Monitoring Performance in the Public Sector, New Brunswick, Transaction Books.
National Educational Research Forum (2000) A National Strategy Consultation Paper, Nottingham, HERF publications.
Nove, A. (1980) The Soviet Economic System, Second edition (London, Allen and Unwin).
Oakeshott, M. (1962) Rationalism in Politics, and Other Essays, London, Methuen.
Polanyi, M. (1959) Personal Knowledge, Manchester, Manchester University Press.
Polanyi, M. (1966) The Tacit Dimension, Garden City, NY, Doubleday.
Pollitt, C. (1990) Managerialism and the Public Services, Oxford, Blackwell.
Popay, J., Rogers, A., and Williams, G. (1998) "Rationale and standards for the systematic review of qualitative literature in health services research", Qualitative Health Research, 8, 3, pp341-51.
Power, M. (1997) The Audit Society: rituals of verification, Oxford, Oxford University Press.
Pring, R. (2000) Philosophy of Educational Research, London, Continuum.
Sackett, D. L., Richardson, , W. S., Rosenberg, W., and Haynes, R. B. (1997) Evidence-Based Medicine: How to practice and teach EBM, New York, Churchill Livingstone.
Sackett, D. L., Rosenberg, W., Gray, J. A. M., Haynes, R. B., and Richardson, W. (1996) "Evidence-based medicine: what it is and what it isn"t", British Medical Journal, 312, pp71-2.
Sargant, N. (1995) "Consumer power as a pillar of democracy", in G. Dench, T. Flower, and K. Gavron (eds) Young at Eighty: the prolific public life of Michael Young, Manchester, Carcanet Press.
Shahar, E. (1997) "A Popperian view of “evidence-based medicine”", Journal of Evaluation in Clinical Practice 3, 2, pp109-16.
Trinder, L. (ed) (2000a) Evidence-Based Practice: a critical appraisal, Oxford, Blackwell Science.
Trinder, L. (2000b) 'Introduction: the context of evidence-based practice', in Trinder 2000a.
Vietor, R. H. K. (1994) Contrived Competition: regulation and deregulation in America, Cambridge MS, Harvard University Press.
1. Davies et al have added two even weaker formulations: 'evidence-influenced' and 'evidence-aware' practice (Davies et al 2000:11).
2. An example of such an account is Sackett et al 1997.
3. Other examples include Oakley's claim that the generalisations of 'well-intentioned and skilful doctors [...] may be fanciful rather than factual [...]'. Indeed, she goes so far as to refer to 'the largely unscientific character of modern medicine', appealing to Illich's notion of 'medical nemesis' (Oakley 2000: 17). Another caricature is Greehalgh's contrast between evidence-based decision-making and 'decision-making by anecdote' (Greenhalgh 1997:4). And Evans, advocating systematic reviews in the field of education, suggests that previously even when health practitioners used research evidence they relied on 'idiosyncratic interpretations of idiosyncratic selections of the available evidence (rather than objective interpretation of all the evidence) [...]' (Evans 2001).
4. Interestingly, Oakley extends this argument to researchers, claiming that 'most research proceeds from the interests of researchers because researchers generally have more to gain from the research process: publications, external research grant income, points in academic research exercises'. She excepts experimental research from this charge, on the grounds that it is 'in a somewhat special category, in that its purpose is to arbitrate between competing interventions, and thus to work towards a solution which is as much (if not more) in the interests of the researched as of the researchers' (Oakley 2000:285). Here, aside from special pleading, there are signs of the anti-academic attitude that underlies much recent promotion of the role of research in relation to policymaking and practice. An even more influential exponent of this was the previous Secretary of State for Education, David Blunkett: see Blunkett 2000; and, for a discussion, Hammersley 2000a.
5. For more extended discussion, see Hammersley 2002.
6. Cochrane often seemed to come close to this view in his advocacy of evidence-based practice in medicine: see Cochrane 1972:30. For a useful discussion of problems that have arisen even where randomised controlled trial methodology has been extensively applied, from a viewpoint that is sympathetic to the movement, see Hampton 1997.
7. It is also important to caution against too narrow an interpretation of that contribution, see Hammersley 1997.
8. In many respects, this model amounts to taking a positivist conception of research as the pattern for all forms of rational action. This perhaps explains why evidence-based practice is sometimes closely related to the idea that professionals should themselves engage in research (see, for example, Hargreaves 1996). This amounts to another version of what I have referred to elsewhere as researcher imperialism, see Hammersley 1993.
9. Oakley is director of the of the Evidence for Policy and Practice Information Co-ordinating Centre (EPPI-Centre), which has been given responsibility for developing systematic reviews in education. See its web-site : http://eppi.ioe.ac.uk/
10. Some recent accounts of the role of research in relation to evidence-based education are more sympathetic to qualitative research: see Davies 1999 and Fitzgibbon 2000. However, it is striking that Fitzgibbon devotes only two paragraphs of her article to discussing it, giving by far the greatest attention to quantitative research. And while Davies 2000b seeks "to redress any such demeaning of qualitative research and evidence" (p292), he fails to recognise the sharp conflict between the orientation of some kinds of qualitative work and quantitative method; and the difficult questions that even more moderate versions of a qualitative approach raise for some of the fundamental assumptions of evidence-based practice; on which see Hammersley 2002:ch4. For an account which takes more account of the distinctiveness of a qualitative approach, see Popay et al 1998.
11. For a discussion of how this review was represented in the mass media in diverse ways, see Hammersley 2000b.
12. I am referring here to the response of the previous Secretary of State for Education, David Blunkett, to the findings of one study of the relationship between homework and achievement; in which he accused the researchers of double standards, of being unwilling to extend to other children the benefits their own children have had. . See S. Farrow "Insulted by Blunkett", The Guardian Wednesday July 21, 1999. In his speech to the CBI Blunkettt also included the following comment: "Some researchers are so obsessed with "critique", so out of touch with reality that they churn out findings which no one with the slightest common sense could take seriously" (Blunkett 1999, quoted in Pring 2000). Mr Blunkett"s response was similar to research which suggested that eviction laws were being applied over-zealously by some local authorities. His comment was: "If this is what our money is going on, it is time for a review of the funding of social research", BBC News web page for 20.11.2000. His criticism of these two studies was repeated in his speech to the ESRC: Blunkett 2000.
13. In the context of evidence-based medicine, Sackett et al talk of "integrating individual clinical expertise with the best available external evidence from systematic research" (Sackett et al 1996). But "integrating" is no better than "uniting": it is a fudge which covers up a serious problem.
14. For the argument that even natural scientific research cannot be reduced to explicit procedures, see Polanyi 1959 and 1966.
15. This is an example of what Oakeshott referred to as "the politics of perfection" Oakeshott 1962:5-6.
16. For the background to this development, see Burnham 1941, Chandler 1977 and 1990.
17. This was closely associated with the rise of management education, and of the accounting profession. The whole process was, of course, simply a further stage in the development of what Marris calls 'managerial' capitalism initiated by the invention of the public, joint-stock, limited liability company in the late nineteenth century (see Marris 1971).
18. As far as I can tell, the arguments for the superiority of the private sector rested on economic theory and anecdotes about 'over-manning', 'bureaucracy' etc. in the public sector; rather than on objective empirical evidence.
19. Performance indicator results have many of the key characteristics of newsworthiness: they are about the Government, they offer apparently clear-cut and authoritative findings, and they often suggest serious failings. For a rather different, but not incompatible, account of the develpments I have sketched here, see Aucoin 1990. Aucoin stresses the role of public choice theory and managerialism, portraying them both as anti-bureaucratic, and argues that there are tensions between the two, especially in relation to centralisation/decentralisation and control/delegation.
20. One aspect of this was criticism of state regulation of industry. For an account of this in the United States, and of the pendulum swings towards and then away from regulation over the course of the twentieth century, see Vietor 1994:312 and passim.
21. This was why the Consumers' Association began to publish Which? magazine: see Sargant 1995:190-2. However, even that publication cannot provide everything one would need to know in order to make the best choice of many kinds of product. The sheer number and diversity of products is simply too large, and subject to too frequent change.
22. It is not clear that quality is given emphasis even where fashion, technical hype, or snobbery will support relatively high prices.
23. Nove writes that in the Soviet Union 'many of the functions of government ministers and senior party officials can be regarded as a form of senior management, in some respects analogous to those of the senior directors of a big Western corporation' (Nove 1980:88). He continues: 'how can the centre issue instructions in a form which does not give rise to ambiguities and contradictions? How can it encourage initiative, new techniques, the flow of desired and undistorted information? By what standards and in what way can we evaluate performance? Around these issues one finds grouped many of the most serious weaknesses of Soviet micro-economics. This is where many things go wrong' (Nove 1980:89). Indeed it is, and not just in the old Soviet Union!
24. In What Works? Davies et al comment that: '[...] the majority of research evidence considered in this text is the output from more formal and systematic enquiries, generated by government departments, research institutes, universities, charitable foundations, consultancy organisations, and a variety of agencies and intermediaries such as the Audit Commission, or Office for Standards in Education (Ofsted)' (Davies et al 2000:3). For a useful discussion of the relationship between research and OFSTED inspection, see Smith 2000.