Cochrane or cock-eyed? how should we conduct systematic reviews of qualitative research?
Paper presented at the Qualitative Evidence-based Practice Conference, Taking a Critical Stance. Coventry University, May 14-16 2001
The quantitative versus qualitative debate has taken significant steps towards reconciliation within the wider field of evidence based practice. Nevertheless, far more insidious discrimination remains. Systematic review methodology exhibits all the characteristics of "institutionalised quantitativism" in that criteria for a "good" review are almost entirely determined by the quantitative methods promoted and perpetuated by the Cochrane Collaboration. Nobody who understands qualitative research would insist that its primary studies demonstrate alien concepts such as "sample size" or "statistical power". Yet comparably fundamental absurdities persist with regard to qualitative syntheses. Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive literature search when concepts such as "data saturation" have an established pedigree? Why should they apologise for an absence of meta-analysis when little-known techniques such as meta-ethnography could be included in a reviewer's toolbox? Why shouldn't they apply systematic, explicit and reproducible principles of thematic or concept analysis to create syntheses that advance our understanding of qualitative issues and highlight research gaps? The author draws on experience of a dozen systematic reviews, a third qualitative, to suggest how systematic reviews of qualitative research might acquire a methodology that is more sympathetic to the paradigm within which they are conducted.
Meta-synthesis, the "science of summing up"(Light & Pillemer 1984), is a building block of evidence based practice. It may be either quantitative (meta-analysis) or qualitative (meta-ethnography). Its value lies in the recognition that busy practitioners find it almost impossible to make decisions based upon the massive and increasing volume of research evidence (Sleep & Clark, 1999). They therefore require an overview of work in a specific area that has been compiled by "finding, evaluating and synthesising evidence from research studies" (Hek, Langton & Blunden, 2000). Historical precedent for the value of such an enterprise dates back at least to Aristotle:
"it is necessary, while formulating the problems of which in our further advance we are to find the solutions, to call into council the views of those of our predecessors who have declared an opinion on the subject, in order that we may profit by whatever is sound in their suggestions and avoid their errors". (Aristotle, De Anima quoted in Bland et al, 1995)
Early experiences of evidence based practice, particularly in the context of critical appraisal, were invariably accompanied by the postscript "But what about qualitative research?". To a certain extent this polarisation into quantitative and qualitative camps has been headed off at every level of the evidence based practice paradigm. Sackett, one of the founding fathers of evidence based healthcare, issued a message of reconciliation through a BMJ Editorial (Sackett & Wennberg, 1997). Similarly Greenhalgh, another leading proponent of the fledgling movement, made a persuasive appeal that "How evidence based is my practice" should be answered in qualitative as well as quantitative terms (Greenhalgh, 1996) and also advocated the importance of Narrative Based Medicine (Greenhalgh, 1999). Within the specific technique of critical appraisal the Critical Appraisal Skills Programme (CASP) in Oxford, among many others, has developed a checklist for appraising qualitative research studies. At the producer, as opposed to consumer, end of the spectrum, recognition by the Cochrane Collaboration of qualitative research was signalled by the formation of a Qualitative Research Methods Network.
Notwithstanding such advances systematic review methodology exhibits all the characteristics of "institutionalised quantitativism" in that criteria for a "good" review are almost entirely determined by the quantitative methods promoted and perpetuated by the Cochrane Collaboration. This appears to be not so much a conscious effort by the Collaboration to discriminate against this particular paradigm of research methodology as a profound lack of awareness of the potential contribution of qualitative methods of meta-synthesis to our knowledge of healthcare interventions. A search of the Cochrane Systematic Review Methodology database in early 1997 yielded only one reference on qualitative systematic reviews - the seminal work Meta-ethnography by Noblit and Hare (1988).
A citation search for the Noblit and Hare reference on the Science Citation and Social Science Citation Indexes via the Web of Science yielded 15 relevant articles. These comprised both methodological discussions and case studies of the use of meta-ethnographic approaches. This led, in turn, to a subject search using the keywords "meta-ethnograph$", "metaethnograph$", "meta-synthesis", and "metasynthesis" on the MEDLINE, CINAHL and Science and Social Science Citation Indexes to retrieve additional articles. This cursory literature search was designed to provide an external frame of reference against which the experience of the author, in conducting four collaborative systematic literature reviews, could be compared. These four systematic reviews, underpinned to varying degrees by the methods described by Noblit and Hare, were as follows:
- A systematic review for the NHS Health Technology Assessment Programme on the Use of Modelling in the Planning and Prioritisation of Clinical Trials (Chilcott et al, In press)
- A systematic review for the NHS Health Technology Assessment Programme on the Use of Health-related Quality of Life Measures in Economic Evaluation (Brazier et al, 1999)
- A systematic review for the English National Board for Nursing, Midwifery and Health Visiting on the nursing contribution to rehabilitation (Nolan et al, 1997)
- A systematic review for the American Association for Retired Persons on coping with multiple chronic health conditions in older persons (Dowzer et al, 2000)
Techniques in meta-ethnography originated within the field of education as a means of synthesising the findings from school inspection reports (Noblit and Hare, 1988). Following identification of a definable set of studies an iterative process of analysis takes place, typically using the using the constant comparative method (Glaser & Strauss, 1967; Kearney 1998a). This leads to the identification of themes or metaphors that permeate the literature (Beck, 2001). The potential of meta-ethnography as a tool for knowledge synthesis was identified at an early stage of its development (Klein, 1989). From education it was a comparatively short migration to the contiguous field of sociology where Zhao (1991) placed such techniques within the broad categorisation of meta-study and its associated branches of metatheory, metamethod and metadata-analysis.
Within the healthcare literature meta-synthesis appears most frequently in nursing research studies (Sherwood, 1999). Occasionally meta-ethnography's origins in education resurface intriguingly in a nursing context (Beck, 2001).
It is interesting to observe clusters of studies around a small number of broad topics:
- Chronic disease (Barroso and Powell-Cope, 2000; Thorne & Paterson, 1998)
- Coping (Burke et al, 1998; Paterson et al, 1998)
- Caring (Beck, 2001; Frederiksson, 1999; Sherwood, 1997)
Ironically the burgeoning interest in the application of the Cochrane Collaboration's systematic reviews methods to the fields of social work, law and education has resulted in the formation of a parallel organisation, the Campbell Collaboration. This in turn has led to a resurgence for meta-ethnography in the education literature, alongside other methods of synthesis such as narrative reviews, vote-counting reviews, meta-analyses and best evidence synthesis (Davies, 2000).
Applications of meta-synthesis within evidence based practice
Qualitative meta-synthesis can make a number of possible contributions to evidence based practice. It can be used:
- For the generation of models and theories (Estabrooks et al, 1994; Finfgeld, 1999; Thorne and Paterson, 1998).
- For the subsequent validation and gauging of the empirical support for theories (Forte, 1998).
- In methodological reviews as a means of determining the existence of certain bodies or "schools" of thought.
- To chart the development of concepts longitudinally in the literature over a period of time (Thorne and Paterson, 1998).
- To complement the findings and interpretation of quantitative systematic reviews, for example in looking at issues around patient acceptance or compliance.
- As a means of identifying significant domains or attributes to enable the development of prototype instruments or scales.
Unlike a quantitative systematic review or meta-analysis where the review process is conducted linearly in a series of discrete, sequential steps and where deviation from an a priori protocol poses threats to validity, the corresponding method in a qualitative review involves continual iteration and retracing of steps. In fact it is arguable whether an a priori review protocol is appropriate to an interpretative process such as meta-ethnography (itself more akin to the process of discovery initiated by a grounded theory approach in primary qualitative research). However for ease of presentation it will suffice to examine each of the stages of the review process for a qualitative review in turn.
Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive literature search when concepts such as "data saturation" have an established pedigree? According to NHSCRD guidelines for conducting systematic reviews of effectiveness (2000):
"The aim of the search is to generate as comprehensive a list as possible of primary studies, both published and unpublished, which may be suitable for answering the questions posed in the review. Identification of relevant literature by a thorough unbiased search strategy is crucial. This is because the validity of the findings of a review is directly related to the comprehensiveness of the literature search that is used to capture the relevant studies."
Here is an example of an instance where the "big is beautiful " approach to systematic reviews required by quantitative reviews is implicitly extended to cover qualitative reviews. Many of the techniques for checking the validity of systematic reviews are underwritten by the fundamental assumption that one is attempting to identify, if not the whole population of studies, as many relevant articles as possible. The funnel plot, for example, is an attempt to quantify the effect that missing studies from this optimal population might have on the results of the review. The intent of the literature search for a quantitative review is to identify a statistically representative sample of the published literature.
Methodological reviews, such as those produced for the NHS Health Technology Assessment Programme, make much of the virtue of the comprehensiveness of the data sources that have been searched. However, such reviews belong more appropriately within a qualitative paradigm. Is it necessary for a qualitative review to be equally comprehensive in its coverage? At this stage it will be appropriate to draw a parallel, not with quantitative systematic reviews, but with primary qualitative research. Statistical representativeness is not a primary requirement for qualitative data collection where systematic non-probabilistic sampling is more commonly used (Mays and Pope, 1996). If we think of the papers in a qualitative review as being 'informants' then we seek to identify specific groups of papers that possess characteristics that are relevant to the phenomenon being studied. We aim to include a wide range of types of papers (i.e. reflecting as many of the themes or schools of thought as possible) and, preferably to select 'key informants' (i.e. papers that lead us to important sources of knowledge in the form of additional citations). The concept of 'data saturation' is also of relevance here. Once a particular theme has been identified further occurrences of this theme are only of interest in strictly quantitative terms unless they expand on or modify an already-identified theme. Interestingly quantitative reviewers are currently seeking methods to define a 'law of diminishing returns' beyond which further literature searching has little justification in order to manage the inordinate expense of the searching process. For qualitative reviews the answer to this problem already exists in the principles of data saturation used in primary studies.
Case study : Systematic review on the Use of Modelling in the Planning and Prioritisation of Clinical Trials
Although the proposal originally submitted to the NHS HTA Programme focused on traditional methods of comprehensive literature searching we also piloted methods of 'citation snowballing' to investigate their potential for methodological reviews. A team of subject experts identified seven key papers representing a number of perspectives on the use of modelling. References at the end of these papers were examined in context and a further 34 papers were identified in this manner. References at the end of these 34 papers led in turn to a further 200 papers. Unfortunately the resources for this project did not allow more than two 'rounds' of this approach - we were unable to follow up the next 'round' of references (n=425). In an ideal world this process would be continued until a point of data saturation is reached i.e. no new references are identified.
Within this context the technique of citation searching is of particular interest. Citation searching, although necessarily complementary to comprehensive literature searching, is viewed with some caution by quantitative reviewers. It can lead to reference or citation bias whereby supporting articles are more likely to be cited and conflicting articles are more likely to be omitted. However, in contrast to the effectiveness literature, where such dangers are likely to be critical, there is little reason to believe that this is equally problematic in the literature of methodology where techniques of argument and counter-argument are essential for the development of ideas and schools of thought. Using the analogy of the 'key informant', however, one might propose that a 'key' article in this context is one that references as broad a selection of seminal articles as possible rather than one which is itself cited by numerous articles! In practice triangulation of both approaches with citation searching (i.e. forward and backwards) will be required.
So does the following statement from the NHS CRD guidelines apply equally to qualitative reviews? :
"The thoroughness of the literature search is one factor that distinguishes systematic reviews from traditional reviews. It is also important to ensure that the process is as thorough and unbiased as possible".
It might be argued that, while it is certainly important for the search process to be free from bias, it is more important that the search process be systematic, explicit and reproducible rather than comprehensive. Thoroughness in this context should apply to the rigour of the search process not to its comprehensiveness. To argue otherwise would most definitely be "cock-eyed".
Literature searching for qualitative systematic reviews should exhibit the following characteristics:
a) Identifying major "schools of thought" in a particular area whilst being alert to the identification of variants, minority views and dissenters. It is particularly important to identify negative or disconfirming cases (Paterson et al, 1998)
b) Searching within a broad range of disciplines so as to bring different views (e.g. clinician, consumer, manager, health economist, statistician, research commissioner etceteras) to bear on the topic in hand.
c) Using complementary electronic and manual search techniques to ensure that materials are not missed either through the inadequacies of indexing or through selective coverage of databases
Techniques in study appraisal are also dominated by the quantitative paradigm. Quantitative systematic reviews run the danger of underestimating the evidence from relevant literature if they only include that of a certain methodological quality (Edwards et al, 1998). This might result in missing or distorting the true message that the review is trying to identify. Perversely, using 'weaker' study designs tends to have the effect of amplifying the strength of a demonstrable effect and so quantitative reviewers are usually willing to trade the loss of additional perspectives against improved precision of findings. Clearly, in a qualitative review, if research is rejected on the basis of design alone there is a very real risk of denying valuable insights that contribute to our interpretation of a phenomenon. Qualitative research, with its relativist perspective, acknowledges the existence of multiple views of equal validity (Popay et al, 1998) and is not readily amenable to the use of the hierarchies of evidence promulgated by evidence based medicine (Barbour, 1998; Barbour, 2001).
Edwards and colleagues (1998) recognise the importance of assessing the message or 'signal' within each individual piece of research. They advocate balancing an assessment of methodological quality against the weight of its message, rather than rejecting studies that fall below a certain quality threshold. They argue that fundamentally flawed research will still be rejected, but other papers will have the importance attached to their signal tempered by the amount of 'noise' around that signal. The balance of these two elements, termed the 'signal to noise ratio', is of great relevance to techniques of meta-ethnography.
Barbour (2001) has recently introduced an important dissenting voice to the stampede to adopt critical appraisal techniques in order for qualitative research to "pass muster" in the new regime of evidence based practice. She describes a number of "technical fixes" (such as purposive sampling, grounded theory, multiple coding, triangulation, and respondent validation) that simply may serve to reassure those that question the contribution that qualitative research might make and yet not advance a broader understanding of the rationale and assumptions behind qualitative research.
With the cautions of Barbour (2001) about the cosmetic overuse of multiple coding still fresh in mind it is perhaps imprudent to assert the need for two observers in qualitative reviews when data extraction and, more importantly, coding of themes takes place (Compare Paterson et al, 1998). Barbour (2001) correctly identifies that "multiple coding concerns the same issue as the quantitative equivalent "inter-rater reliability" and is a response to the charge of subjectivity sometimes levelled at the process of qualitative data analysis". On the surface it appears another example of qualitative reviewers having to play by the rules pioneered by quantitative reviewers. However to what extent is the use of multiple coders a property of the material being reviewed and how much is it a function of essential systematic review methodology?
Here the answer to our question does not seem to lie in the fact that both quantitative and qualitative reviewers have arrived at a common device of multiple observers. Rather its importance lies in why it is being done. In quantitative reviews the aim is to achieve an acceptable degree of concordance between researchers, thereby increasing confidence in the validity of the review. This is usually assessed using the device of a kappa statistic. However as Barbour (2001) again observes, admittedly in the context of primary qualitative research:
"the degree of concordance between researchers is not really important; what is ultimately of value is the content of disagreements and the insights that discussion can provide for refining coding frames. The greatest potential of multiple coding lies in its capacity to furnish alternative interpretations and thereby......in alerting researchers to all potentially competing explanations. Such exercises encourage thoroughness, both in interrogating the data at hand and in providing an account of how an analysis was developed. Whether this is carried out by a conscientious lone researcher, by a team, or by involving independent experts is immaterial: what matters is that a systematic process is followed and that this is rendered transparent in the written research project".
The use of multiple observers in study analysis for qualitative reviews, such as was the case in our comorbidity review, should be clearly defined as being for the unique requirements of qualitative research. It should not be misinterpreted as an attempt to adopt uncritically the model promulgated by quantitative reviews of effectiveness.
The 'signal to noise' metaphor from the field of communication (mentioned above) is also pertinent to debates concerning whether one should conduct meta-ethnography of authors' own interpretations of data or only of verbatim comments reported by authors in their texts. It is possible to see the two sorts of data, that is:
- Primary data from transcripts, and
- Secondary data from published articles
as being complementary sources of a 'signal'. Primary data has less potential for noise but is of such large quantities that it is very difficult to pinpoint the exact nature of a 'signal' while secondary data is much more selective, is amplified by the authors' interpretations but therefore runs the risk of losing integrity. Interestingly such issues are analogous to debates about individual patient data versus literature (Stewart & Parmar, 1993) in the meta-analytic domain.
Thematic analysis can be characterised within two approaches:
A) An approach analogous to grounded theory (Wuest 2000) is used whereby the constant comparative method produces insights from the literature that are 'discovered' rather than being a preconceived framework used by the researcher.
B) An existing organising construct is used to provide meaning to the disparate themes identified from the analysis (Frederiksson, 1999).
The reviews outlined in this paper include both approaches to thematic analysis; the comorbidity review used a grounded theory approach to discover approaches to coping as documented in the literature while the rehabilitation review used a conceptual framework by Corbin and Strauss (1988) as a means of structuring themes according to the characteristics of the target conditions.
According to Sandowski and colleagues (1997) the accumulation of qualitative studies in recent years has led to little cumulation of the understandings gained from them. Techniques used have included reciprocal translations of key metaphors and concepts and qualitative and quantitative comparative analyses to produce narrative and theoretical integrations. The major problem, yet to be resolved, is developing usable and communicable systematic approaches that synthesise composite findings and yet maintain the integrity of individual studies.
Once the synthesis has been completed follow-up validation strategies may be required to safeguard the validity of the findings (Mays and Pope, 1996). Several techniques are available and some of these have been employed in the reviews outlined in this paper:
a) "Triangulation", comparing oral testimony from subjects with the written record of the systematic review, was used in the review for the American Association for Retired Persons. Phase 2 of this project involved eliciting data from those experiencing multiple chronic health conditions to see if their perceptions matched the framework derived from the literature.
b) The framework developed from the methodological literature from the review on the use of modelling in trials was validated by the random selection of case studies to be assessed against this framework
c) A checklist derived from the literature of the use of health-related quality of life measures in economic evaluation was validated by sampling from a single year of the health economics literature.
d) Findings from the general literature on the role of the nurse in rehabilitation were then examined using theoretical sampling in conditions that were characterised according to a two-by-two matrix of acute versus chronic and young-onset versus old-onset.
Other NHS Health Technology Assessment reviews have attempted to validate constructs from the literature in interviews or with focus groups. From the broader literature Khunti (1999) describes how multiple methods (systematic literature searching, brainstorming with health professionals and patients, a focus group and interviews with key informants) can be used for hypothesis generation. It is interesting in this context to note that the literature review identified 37 out of 54 putative factors (68.5%) affecting the quality of care of patients with diabetes. A combination of brainstorming and literature review identified 51 out of 54 factors (an impressive 94.4%).
Meta-study permits critical analysis of the location of current inquiry within the larger context of systematic patterns in knowledge development (Thorne and Paterson, 1998). It is important to note, however, that the sheer quantities of data involved may result in certain arbitrary decisions being made. For example Thorne and Paterson (1998) only examine in depth one of a large number of themes identified while it is not unusual (in common with primary qualitative research) for researchers to report only the "top themes" identified from the literature which may result in the loss of important concepts.
In addition to detailed exposition of multiple themes and in-depth analysis of those regarded as being specifically important it is essential that the qualitative reviewer indicates the existence of negative or disconfirming cases (Paterson et al, 1998). This is particularly the case in methodological reviews where a minority approach may have tremendous potential for development but be otherwise overshadowed by the existence of a substantive corpus of work. This is analogous to the importance of investigating sources of homogeneity in quantitative meta-analyses (Thompson, 1994).
From the brief exploration above it is possible to identify some of the defining characteristics of qualitative systematic reviews:
- Their intention is not to identify all literature on a particular topic. Whilst the aim is to identify specific groups of papers that possess characteristics that are relevant to the phenomenon being studied this need not imply statistical representativeness.
- Particular attention is focused on negative or disconfirming cases. This adds to the richness of the insight that the review provides on the phenomenon of interest.
- Such reviews acknowledge the existence of multiple views. Caution must therefore be exercised so that valuable insights are not lost by application of some a priori definition of quality through critical appraisal.
- Although use of multiple reviewers may be in common with quantitative reviews the essential contribution of this to a qualitative review lies in its capacity to furnish alternative interpretations, thereby alerting researchers to potentially competing explanations.
- Whereas quantitative reviews seek to handle "noise" by exclusion (through the application of a predetermined quality threshold) qualitative reviews seek to optimise the "signal to noise" ratio.
- Following on from (5) it will often be necessary to validate the findings from a qualitative review with reference to a subset of case studies, use of key informants or focus groups or other methods of triangulation.
The Way Forward?
As qualitative methods become popular and qualitative reports abundant, it is a sign of methodological maturity when researchers discuss techniques for synthesising findings about related phenomena from diverse samples (Kearney, 1998b). Advocates of evidence based practice are starting to look beyond the more established models of research synthesis for innovative approaches (Greenhalgh, 1998; Sleep & Clark, 1999). Qualitative metasynthesis is one response to concerns about the relevance and utility (Sandelowski, 1997; Sandelowski et al, 1997) of qualitative research, and much work remains to be done in developing and harnessing appropriate methodologies. Given the pioneering role of the Cochrane Collaboration in developing review methodology it is doubly unfortunate that opportunities to develop qualitative systematic reviews are constrained by:
- the implicit assumption that systematic review methods that are conceived and developed in the context of quantitative reviews should be translated uncritically for use in qualitative reviews.
- the apparent willingness of the Cochrane Collaboration Qualitative Research Network to adopt a "handmaiden" rather than "partnership" role in contributing to methodological developments.
This latter constraint is best exemplified by an extract from the objectives from the draft module of the proposed Qualitative Research Methods Group:
- Develop and disseminate appropriate methodological standards for combining data from qualitative studies with other data within the context of a systematic review.
There is currently no forum for the development of qualitative systematic reviews per se (as opposed to the use of qualitative data within systematic reviews). This deficiency is shared by methodological reviews that seek to embody existing schools of thought or isolated viewpoints in synthetical documents that advance the state of our knowledge. Paradoxically this includes documents, such as those produced by the NHS Health Technology Assessment Programme, that inform the conduct of quantitative systematic reviews! Now that is what I call cock-eyed!
Barbour RS (1998) Mixing qualitative methods: quality assurance or qualitative quagmire? Qualitative Health Research 8: 352-361
Barbour RS (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 322: 1115-1117.
Barroso J & Powell-Cope GM (2000) Metasynthesis of qualitative research on living with HIV infection. Qualitative Health Research 10 (3): 340-353.
Beck CT (2001) Caring within nursing education: A metasynthesis. Journal of Nursing Education 40 (3): 101-109.
Bland CJ, Meurer LN & Maldonado G (1995) A systematic approach to conducting a non-statistical meta-analysis of research literature. Academic Medicine 70 (7): 642-653.
Brazier, J., Deverill, M., Green, C., Harper, R. & Booth, A. (1999). A review of the use of health status measures in economic evaluation. Health Technology Assessment 3 (9).
Burke SO, Kauffman E, Costello E, Wiskin N & Harrison MB (1998) Stressors in families with a child with a chronic condition: an analysis of qualitative studies and a framework. Canadian Journal of Nursing Research 30(1): 71-95.
Chilcott J, Brennan A & Booth A (In press) The Use of Modelling in the Planning and Prioritisation of Clinical Trials. Health Technology Assessment
Corbin JM and Strauss A (1988) Unending work and care: managing chronic illness at home. Jossey-Bass, San Francisco.
Davies P, Wolf FM & Holmes L (2000) An Organisational Framework for Preparing and Maintaining Systematic Reviews in Education: A Discussion Document. http://campbell.gse.upenn.edu/papers/9a_framework.html [accessed 01/05/2001]
Davies P (2000) The relevance of systematic reviews to educational policy and practice. Oxford Review of Education 26 (3-4): 365-378.
Dowzer C, Basford L, Booth A & Poon, LW (2000) Living with Multiple Health Conditions: a systematic search and literature review. [A report for the American Association of Retired Persons]. Sheffield: School of Health and Related Research (ScHARR), University of Sheffield.
Edwards AG, Russell LT & Stott NCH (1998) Signal versus noise in the evidence base for medicine: an alternative to hierarchies of evidence? Family Practice 15 (4): 319-322
Estabrooks CA, Field PA & Morse JM (1994). Aggregating qualitative findings: An approach to theory development. Qualitative Health Research 4: 503-511.
Finfgeld DL (1999) Courage as a process of pushing beyond the struggle. Qualitative Health Research 9 (6): 803-814.
Forte JA (1998) Power and role-taking: a review of theory, research, and practice. Journal of Human Behavior in the Social Environment 1(4): 27-56.
Fredriksson L (1999) Modes of relating in a caring conversation: a research synthesis on presence, touch and listening. Journal of Advanced Nursing 30(5): 1167-76.
Glaser B & Strauss A (1967) The discovery of grounded theory. Chicago: Aldine.
Greenhalgh, T (1996) "Is my practice evidence-based?" BMJ 313: 957-958.
Greenhalgh T (1998) Meta-analysis and beyond: applying secondary research methods to primary care. British Journal of General Practice 48 (433): 1540-1541.
Greenhalgh, T (1999) Narrative based medicine in an evidence based world. BMJ 318: 323-325.
Hek G, Langton H & Blunden G (2000) Systematically searching and reviewing literature. Nurse Researcher 7 (3): 40-57.
Jensen LA & Allen MN (1994) A synthesis of qualitative research on wellness-illness. Qualitative Health Research 4: 349-69.
Jensen L & Allen M (1996) Meta-synthesis of qualitative findings. Qualitative Health Research 6: 553-560.
Kearney MH (1998a) Truthful self-nurturing: a grounded formal theory of women's addiction recovery. Qualitative Health Research 8 (4): 495-512.
Kearney MH (1998b) Ready-to-wear: Discovering grounded formal theory. Research in Nursing & Health 21 (2): 179-186.
Klein SS (1989) Research and Practice - Implications for Knowledge Synthesis in Education. Knowledge-Creation Diffusion Utilization 11(1): 58-78.
Khunti K (1999) Use of multiple methods to determine factors affecting quality of care of patients with diabetes. Family Practice 16(5): 489-94
Lemmer B, Grellier R & Steven J (1999) Systematic review of nonrandom and qualitative research literature: exploring and uncovering an evidence base for health visiting and decision making. Qualitative Health Research 9(3): 315-28.
Light RJ & Pillemer DB (1984). Summing up: the science of reviewing research. Cambridge, MA: Harvard University Press.
Mays N & Pope C (1996) Qualitative research in health care. London: BMJ Publishing Group.
NHS Centre for Reviews and Dissemination (2000). Undertaking Systematic Reviews of Research Effectiveness: Guidelines for those carrying out or commissioning reviews. Report 4. 2nd edition. York: NHS Centre for Reviews and Dissemination, University of York.
Noblit GW & Hare RD (1988) Meta-ethnography: synthesising qualitative studies. (Qualitative Research Methods, Volume 11) London: Sage.
Nolan M., Booth A. & Nolan J (1997) New directions in rehabilitation: exploring the nursing contribution. Research Reports series No. 6. London: English National Board for Nursing, Midwifery and Health Visiting.
Paterson BL, Thorne S & Dewis M (1998) State of the science. Adapting to and managing diabetes. Image - the Journal of Nursing Scholarship 30(1): 57-62.
Popay J, Rogers A & Williams G (1998). Rationale and Standards for the Systematic Review of Qualitative Literature in Health Services Research. Qualitative Health Research 8 (3): 341-351.
Sackett DL and Wennberg JE (1997) Choosing the best research design for each question. BMJ 315: 1636.
Sandelowski M (1997) To be of use: Enhancing the utility of qualitative research. Nursing Outlook 45: 125-132.
Sandelowski M, Docherty S & Emden C (1997) Qualitative meta-synthesis: Issues and techniques. Research in Nursing & Health 20: 365-371.
Sherwood GD (1997) Meta-synthesis of qualitative analyses of caring: defining a therapeutic model of nursing. Advanced Practice Nursing Quarterly 3(1):32-42.
Sherwood G (1999) Meta-synthesis: merging qualitative studies to develop nursing knowledge. International Journal for Human Caring 3 (1): 37-42.
Sleep J & Clark E (1999) Weighing up the evidence: the contribution of critical literature reviews to the development of practice. NT Research 4(4): 306-14.
Smith M & Stullenbarger E (1991) A prototype for integrative review and meta-analysis of nursing research. Journal of Advanced Nursing 16(11): 1272-83.
Stewart L & Parmar MKB (1993) Meta-analysis of the literature or of individual patient data: Is there a difference? Lancet 341: 418-422.
Thompson SG (1994) Why sources of heterogeneity in meta-analysis should be investigated. British Medical Journal 309: 1351-1355.
Thorne S & Paterson B (1998) Shifting images of chronic illness. Image - the Journal of Nursing Scholarship 30(2):173-8
Wuest J (2000) Negotiating with helping systems: An example of grounded theory evolving through emergent fit. Qualitative Health Research 10 (1): 51-70
Zhao SY (1991) Metatheory, Metamethod, Meta-Data-Analysis - What, Why, and How. Sociological Perspectives 34 (3): 377-390.