Education-line Home Page

Embedding Peer Review of Teaching into Departmental Practice.

Clare Kell

Department of Physiotherapy
Cardiff University

Paper presented at the British Educational Research Association Annual Conference, University of Glamorgan, 14-17 September 2005

Email: kell@cardiff.ac.uk
Tel: 029 2074 2267
Address for correspondence:
Clare Kell
Department of Physiotherapy
Cardiff University
Ty Dewi Sant
Heath Park
Cardiff
CF14 4XN
UK

BACKGROUND AND PURPOSE

In response to calls from the Dearing Report (NCIHE, 1999), Quality Assurance Agency and former ILTHE, Higher Education Institutions (HEIs) have been keen to demonstrate the effectiveness of student learning experiences. Focus on the student learning experience has placed emphasis on the teaching environment within which the learning is facilitated. Teaching staff within HEIs have been given the opportunity to internally monitor the students’ learning experiences provided that reports are used to inform any self-reflection document required by auditing bodies (QAA, 2000). Many HEIs operate the dual systems of student evaluation and feedback on their learning experiences, and colleague / peer review or observation of individual learning events or experiences to support their reflective processes. These latter peer-led sessions have formed the basis of much discussion both in the literature and in local academic departments, with rationale ranging from audit / judgmental purposes (QAA, 1996; Shortland 2004), to those focussed on dialogue for mutual benefit and enhancement (Martin and Double, 1998; Orland-Barak, 2005).

The Department of Physiotherapy, Cardiff University (formally the University of Wales College of Medicine) piloted a form of Peer Review of Teaching (PRT) in 1999, based in part on the work of Marsh (1987). This first approach to PRT required staff to be paired (non-reciprocally) within their subject teaching teams. No pre-observation meeting took place and the ethos of the observation was a ‘spot check’ of teaching performance. Post observation dialogue was intended to be self-reflective for both parties and was supported by session-specific feedback from the students. Within 2 years the PRT process had run its course; departmental staff were engaging in a wide-range of informal teaching and learning discussions and felt that there was no longer any place for an audit-based process.

During 2001 the Department piloted a peer triad approach (Gosling, 2000). The Peer Triad process enabled small groups of staff to be drawn from across the subject curriculum so that session content became less important. Each Triad was encouraged to select an annual theme and to use the PRT process to both ‘unpack’ each other’s teaching rationale (MacKinnon, 2001), and to engage with specific pedagogic innovations. Triads met as frequently as they wished and each was asked to provide an anonymous summary of observations, thoughts for wider dissemination and possible development needs. Formative PRT proforma continued to help frame the observations: these documents were confidential to the Triad. Feedback suggested that the Triad approach was beneficial to individual members of staff, helped staff explore the published evidence for their innovation and empowered staff to value their ideas and discussions.

In response to NHS workforce planning directives, undergraduate physiotherapy intake numbers more than doubled between 2001 and 2003 with staff recruitment to match; many of the new academic staff were contracted part time to retain NHS service. The Departmental Quality Assurance report of 2003/04 recorded a 67% completion rate for PRT. This low return was unexpected and unprecedented. In response, the Department embarked on a detailed overview of both the ‘new’ Departmental context and the benefits or otherwise of the existing PRT process. This paper reports the qualitative processes that mapped perception of PRT and resulted in the development of a new approach better suited to the changing culture and staff within the department.

METHODOLOGY

All staff were invited to a PRT workshop in protected staff-development time. Staff selected pseudonyms and used these to anonymously, but traceably, respond to the following data collection activities.

Perception of terms:

Blackwell et al (2001) and Gosling (2002) recommend that departments help staff verbalise their perceptions about a PRT process in order to make overt and then clarify issues of urban myth and reality. Using pseudonyms to track responses, participants were asked to reflect on their personal interpretation of the terms: peer, review, teaching and write definitions for each. From these 3 word-lists staff were asked to select one combined definition that best described their perception of what PRT meant for them. Individuals were then randomly paired and asked to discuss their individual definitions before compiling a consensus PRT definition. This final definition carried the pseudonyms of both staff so that the researchers could consider how far each participant had moved from their initial perception in order to generate the consensus.

Reflection about the existing PRT process:

In a paper discussing perceptions of PRT processes, Gosling (2002) offered a table of keywords / phrases to compare three different PRT models (evaluation model, development model, peer review model). Keywords are presented to reflect how each model addresses purpose, outcome, confidentiality issues etc. Each participant was issued with 2 differently coloured pens and asked to 1) circle keywords on the template that best reflected their perception of the existing departmental PRT process and 2) to use another colour to circle those keywords that would reflect their ‘Utopian ideal’ for a PRT process. Responses were again identified using the pseudonym.

Group reflection and discussion:

The activities above were intended to help individual staff reflect on their perceptions of the value and purpose of PRT processes and to consider their personal roles within the existing departmental system. The final activity was an opportunity for staff to verbalise their concerns and discuss in the safety of others. Groups of 4 self-selected a theme for the discussion and used a storytelling, action-learning approach to help one group member explore relevant personal experiences in detail. Each group, again using their pseudonyms, produced a list of emergent key issues.

Data processing:

All handwritten transcripts were sealed and passed to a third party for transcription to ensure that anonymity was preserved.

Data analysis:

A panel of four academic staff reviewed the transcribed data. Independent reviewers generated themes, which through discussion within the panel, led to the creation of templates for each of the ‘definition’ activities. The Gosling (2002) activity was analysed using keyword response frequencies.

RESULTS

78% (n=20) of the Departmental academic staff participated in the workshop. All data was usable and traceable.

Perception of terms:

Table 1 presents the responses to the individual definition activity.. The majority of staff viewed PRT as a reflective process between colleagues that could be used to facilitate the development of the students’ learning experience. 10 of the 44 statements considering the term ‘review’ included elements of a judgmental, summative and audit-like process operated within a ‘power’ dimension.

Key word

Definitions: Emergent themes

(examples of exact descriptors in brackets)

Total number of statements per theme

Peer…..

Colleague
(colleague, fellow)

Power match / mismatch
(equal, supportive, senior-led)

18
 

18

Teaching….

Methods
(techniques, giving info, facilitation methods)

Purpose
( Assisting learning, personal development)

25
 

34

Review…..

Purpose:

  • a) Formative

  • b) Summative

  • How?

  • a) Tools (audit)

  • b) Interaction (discuss)

  • c) Personal (reflect)

  • d) Affective
  •  

    4
    9

     

    10
    8
    23
    3

    PRT…..

    Methods:
    (revisit, qu and evaluation, assessment)

    Purpose:
    (teaching community, learning development)

    Affective:
    (positive support, respect)

    21


    6

    6

    Table 1: Response frequencies to themes emerging from individual ‘definitions;’ activity.

    Responses from the paired discussion of terms generated similar themes for each term to those presented in Table 1. The notable exception was the disappearance of all words related to audit and summative assessment.

    Perception of existing process v utopian ideal.

    Table 2 presents the statements from the Gosling (2002) table that were circled. There were a wide range of descriptors used to describe perceptions about the existing departmental PRT process; some staff considered the existing process to be summative and judgemental in nature. Descriptors for a ‘Utopian’ Ideal were more focussed and reflected a formative process being used to promote a culture of learning and teaching discussion and development.

    Group discussions.

    Staff formed 5 small groups for the discussion activity. All groups discussed issues related to PRT methods and the affective elements perceived to be involved. Written feedback included the following points:

    1. All groups thought the process title should be changed
    2. Any PRT process resulting from the review day should be time efficient and time protected
    3. The process should be supported at all levels with adequate and appropriate training needs met
    4. Observation / discussion should occur across the range of teaching contexts and teaching-role activities
    5. A need to help staff develop skills in personal reflective practice and reflective dialogue.
    6. A unanimous need to overtly acknowledge and address the affective components of PRT
    7. Overwhelming support for an empowering, meaningful process that could be streamlined into CPD and other developmental and enhancement activities.

    Results summary:

    The data collection process, while innovative, was robust and produced responses that were considered to be a true representation of the views of the departmental staff. The results identified a lack of consensus / clarity in terms of rationale, purpose and intended ethos of the existing PRT process. There did appear to be consensus about what a PRT process could achieve. Affective and value statements were a common feature throughout the results.

    Table 2: Perceptions of existing process v utopian ideal

    Statements listed were circled by 10 or more (max 20) members of staff.

    * Indicates statements circled by 15 – 17 staff,

    ** indicates statements circled by 18 – 20 staff

     

    Existing process

    Utopian ideal

    How?

    Teachers observe each other *

    Teachers observe each other *

    Purpose?

    Identify under-performance, QA, assessment, appraisal

    Demonstrate competency

    Engagement in discussion about teaching / self and mutual reflection

     

    Demonstrate competency

    Engagement in discussion about teaching / self and mutual reflection **

    Outcome?

    Report / plan

    Discussion, wider experience of teaching methods *

     

    Discussion, wider experience of teaching methods **

    Status of evidence?

    Peer shared perceptions

    Peer shared perceptions *

    Relationship?

    Equality / mutuality

    Equality / mutuality **

    Confidentiality?

    Between observed and examiner

    Shared within learning set

     

    Shared within learning set *

    Inclusion?

    Includes all staff

    Includes all staff **

    Judgment?

    How to improve

    Non-judgmental, constructive feedback

     

    Non-judgmental, constructive feedback **

    What is observed?

    Teaching performance, class materials

    Teaching performance, class materials *

    Beneficiaries?

    Mutual benefit between peers *

    Mutual benefit between peers *

    Benefits institution *

    Conditions for success?

    Teacher is valued

    Teacher is valued **

    Risks?

    Risk of complacency, conservatism

     

    DISCUSSION

    Having asked for staff to engage in an honest, open yet time-consuming data collection process, the researchers felt an ethical obligation to involve colleagues in the development of their next PRT process. The following sections therefore, present some of the issues that were discussed with the academic staff and management team.

    Ownership.

    Giving staff ownership of the PRT process was seen as recognition of the autonomy with which academic staff carry out many of their roles (Shortland, 2004) and likely to result in the development of a flexible process that could be tailored to individual / local contexts and needs (Blackwell and Preece, 2001).

    For example, Shortland (2004) deconstructed a university’s PRT documentation and combined the results with direct observation of a PRT triad. The observations occurred over a 2-year period either side of a QAA Institutional Review. Her findings suggested that, when ownership for the PRT process lay with the triad (post- QAA review), staff were empowered to adapt the central policy to suit their own needs (while continuing to meet management’s request for form-filling p. 227). The triad staff recognised the benefits, to themselves as individuals and members of a discussion group, of engaging in a PRT process – the staff shared a ‘new agenda of professional development’ (p. 227).

    Rationale and purpose.

    The data collected for this study suggested that there was a mixed perception among the staff about the PRT process’ purpose and rationale: established staff saw the process as formative and useful for personal, professional development; newer staff viewed the process as summative and audit-like. While all academic staff within the department of Physiotherapy, CU are state registered physiotherapists, recent changes within the NHS sector have increased focus on competency to practice. As such, practising therapists are regularly reviewed by their line-managers. To newer academic staff any process involving the observation of practice was thought therefore to be summative, audit-like and seniority- / management- led. While efforts had been made to integrate newer staff into the departmental PRT process, established staff were unaware of this perceptual difference. The discussions that followed presentation of the study results helped all staff to discuss and clarify issues of process rationale and purpose.

    Both management and academic staff wanted a process that would help assure the quality and efficiency of student learning experiences (Clegg et al, 2002). All parties wanted focus to be taken away from content / stand-up performance and instead embrace the breadth of the ‘teaching’ role and its impact on the total learning environment (Radloff, 1998; Hammersley-Fletcher and Orsmond, 2005). Academic staff (established and new) agreed that PRT, and the ensuing reflective dialogue between peers, to be beneficial both to their personal professional development and teaching development across the whole department (Shulman, 2002). Ethos:

    The study results exposed a lack of clarity about the ethos of the existing PRT process. While descriptions of an ‘Utopian’ Ideal were fairly consistent, the existing process was viewed as formative by some and summative by others. Informal discussions with staff who had not engaged with the PRT process in 2003/04 generated two main themes: lack of time, and a previous review that was so judgmental and ‘cloning’ that they had refused to take part again – even though triad groups had changed. Keeping identities confidential, this latter problem was discussed with the staff. All staff worried about having a similar experience.

    Many authors advise that reflective practice for the mutual benefit of both the reviewer and reviewed is an essential component of any PRT process designed to enhance and value learning and teaching diversity (Hammersley-Fletcher and Orsmond, 2004; Beaty, 1998). Reflective practice based dialogue, occurring in an environment of trust and mutual ownership, will help both parties ‘unpack’ their practises as instinctive teachers (MacKinnon, 2001) and go beyond the assumptions both take for granted in their approaches to teaching (Pill, 2005; Paris and Gespass, 2001).

    All state registered physiotherapists are required to undertake Continuing Professional development (CPD) activities to maintain their registered status. Reflective practice is a core ethos of these activities. While staff had experience supporting the development of reflective practice skills in their undergraduate students, many were concerned about the skills required to facilitate peer reflection (Hammersley-Fletcher and Orsmond, 2005). The researchers were requested to append examples of reflective practice models, sample questions and key ‘giving formative feedback’ points to any new process paperwork. Management also agreed to support a staff development afternoon considering issues related to the giving and receiving of verbal feedback.

    Affective issues:

    The data collection discussions had raised staff concerns about affective issues related to any PRT process. Even within a process that focussed on mutual development, there was concern for ‘pulling of rank’, ‘undercurrents of power games’ etc. Overt discussion of these concerns led staff to agree that the term ‘Peer’ referred to an equal with respect to learning and teaching activity i.e. although one member of staff may be more administratively senior to their triad, all were equal in the functions of teaching that they performed (Gosling, 2003). The term ‘Review’ was felt to be unhelpful with its judgmental undertones for practising clinical staff. Staff selected ‘Reflection’ to describe the new process acknowledging the trust (Allen, 2002) and reciprocal collaboration (Martin and Double, 1998) at its foundation.

    Data collection:

    Shortland (2004) describes the common mismatch between rhetoric and practice when staff are asked to evidence the observations they conduct e.g. formative rhetoric supported by closed question, ranking proforma. Staff discussed some common options (Shortland, 2003; Gosling, 2002; MacKinnon, 2001), the quality of data each was likely to produce and the influence that this data might play on the quality of the reflective practice that was possible (Pill, 2005). It was agreed that the new process would encourage staff to use a variety of data /evidence collecting options to best suit the needs of the individuals and potential triad innovations. Data collection transcripts would continue to be confidential to the parties directly involved with central returns being required for completion data only. Triads are asked to feed interesting practice issues and possible suggestions for role support and development into the programme review processes. These returns could be anonymous.

    Grounded in practice:

    All staff agreed that, while developing a new PRT process was essential, nothing would change if the policy could not be embedded into every day practice (Clegg, 2003). While staff didn’t want the process to be seen as an additional chore (Orland-Barak, 2005), there was consensus that a system relying too heavily on the existing subject team structure could quickly stagnate the conversations and potential innovations. Many staff felt that previous systems had benefited from triads drawn across subject and year group teaching. Staff agreed that, with the inclusion of a system for negotiation, the existing process of allocation to triads worked well and should be continued. Triads would operate for two full academic cycles.

    Relevance to practice: the new Peer Reflection of Teaching process.

    The data collection activities and ensuing discussion and staff development events have developed a department-wide clarity of purpose and intent. The ‘new’ PRT process is a change from what could have been perceived as a judgemental review of teaching skills and abilities, to that of a mutually beneficial constructive reflection. Observers are invited into sessions to support and facilitate the reflection process. While the focus of the data collection could be teaching performance, staff are requested to view the range of their teaching role as it impacts on both student learning and the student-learning environment. Even though the process commenced in January 2005, 4 out of 7 triads have engaged with the process and report some excitement with the activities and outcomes they have explored. Initial conversations suggest that ‘time’ is the main barrier for other staff – an issue to be addressed during the next academic cycle.

    CONCLUSION

    This study was established in response to an unexpectedly low PRT return. The Quality Assurance audit exposed an underlying communication and induction problem between existing and newly recruited members of staff. While the department prided itself on an established commitment to and engagement with PRT, it was guilty of policy-holding complacency. This study has demonstrated the problems associated with assumed cultural practice and socialisation. Undergraduate BSc (Hons) Physiotherapy programmes are subject to quinquennial review by professional bodies. PRT will now be included in the review process to ensure that systems are relevant, aligned and operable. The Department has learnt the difference between policy-holding and embedded cultural practice (Boud, 1999).

    References

    Allen, L. (2002) ‘Consenting adults in private’ – Union and management perspectives on peer observation of teaching, LTSN Generic Centre; Feb 2002.

    Beaty, L. (1998) The professional development of teachers in Higher Education: structures, methods and responsibilities, Innovations in Education and Training International, 35, 99-107.

    Blackwell, R., Channell, J. & Williams, J. (2001) Teaching circles: a way forward for the part-time teachers in higher education, The International Journal for Academic Development, 6, 40-53.

    Blackwell, R. & Preece, D. (2001) Changing Higher Education, The International Journal of Management Education, www.business.heacademy.ac.uk/publications/journal/vol1no3paper1/blackwell.pdf

    Boud, D. (1999) Situating academic development in professional work: using peer learning, International Journal of Academic Development, 4, 3-10.

    Clegg, S., Tan, J. & Saeidi, S. (2002) Reflecting or acting? Reflective practice and continuing professional development in Higher Education, Reflective Practice, 3, 131-146.

    Clegg, S. (2003) Problematising ourselves: continuing professional development in Higher Education, International Journal for Academic Development, 8, 37-50.

    Gosling, D. (2000) Guidelines for peer observation of learning and teaching, LTSN: ESCALATE regional networking seminars May-Oct 2000.

    Gosling, D. (2002) Models of peer observation of teaching, LTSN: Generic centre.

    Gosling, D. (2003) ESCALATE report: Research project on peer observation of teaching, www.escalte.ac.uk/resources/peerobservation2 .

    Hammersley-Fletcher, L. & Orsmond, P. (2005) Reflecting on reflective practices within peer observation, Studies in Higher Education, 30, 213-224.

    Hammersley-Fletcher, L. & Orsmond, P. (2004) Evaluating our peers: is peer observation a meaningful process? Studies in Higher Education, 29, 489-503.

    MacKinnon, M. M. (2001) Using observational feedback to promote academic development, The International Journal for Academic Development, 6, 21-28.

    Marsh, D. (1987) Staff Appraisal: The model of self-evaluation, In Edwards, J. (1991) Evaluation in Adult and Further Education (Liverpool, The Worker’s Educational Association, p102).

    Martin, G. & Double, J. (1998) Developing Higher Education teaching skills through peer observation and collaborative reflection, Innovations in Teaching and Training Interactions, 35, 161-170.

    National Committee of Inquiry into Higher Education (1997) Higher Education in the Learning Society (The Dearing Report), HMSO, London.

    Orland-Barak, L. (2005) Portfolios as evidence of reflective practice: what remains ‘untold’, Educational Research, 47, 25-44.

    Paris, C. & Gespass, S. (2001) Examining the mismatch between learner-centred teaching and teacher-centred supervision, Journal of Teacher Education, 52, 398-412

    Pill, A. (2005) Models of professional development in the education and practice of new teachers in higher education, Teaching in Higher Education, 10, 175-188.

    Quality Assurance Agency for Higher Education: QAA (2000) The framework for Higher Education quality in England, Wales and Northern Ireland, QAA: London.

    Radloff, P. (1998) Do we treat time and space seriously enough in teaching and learning? In Black B and Stanley N (eds) Teaching in changing times. Proceeds of the 7th Annual Teaching Learning Forum, University of Western Australia, Feb 1998.

    Shortland, S. (2003) Observing teaching in HE: A case study of classroom observation within peer observation, www.business.ltsn.ac.uk/resources/reflect/conf/2003/shortland

    Shortland, S. (2004) Peer observation: a tool for staff development or compliance? Journal of Further and Higher Education, 28, 219-228.

    Shulman, L. S. (2002) Making differences: a table of learning, Change, Nov/ Dec: 37-44.

    This document was added to the Education-Line database on 20 October 2005