Paper presented at the European Conference on Educational Research, University of Hamburg, 17-20 September 2003

May be copied for educational use provided that the source is acknowledged

The importance of VLEs in European Higher Education has grown in the last five years under the influence of national developments and European policy initiatives. These changes have been accompanied by, and extended through the initiatives of multinational corporations based in North America, although a number of national products, including open source VLEs are available.

This situation is described in the EU e- learning strategy 'Virtual campuses for all students'. "By end 2005, Member States, supported by the eLearning and eTEN programmes should ensure that all universities offer online access¹ for students and researchers to maximise the quality and efficiency of learning processes and activities."²

The cost structure of the products of the major commercial suppliers tends to encourage central provision of VLE and, it is argued, a degree of inflexibility in delivery especially when linked to student management information systems. [Also known as a Managed Learning Environment - MLE]

It is also argued that this approach promotes a degree of pedagogical inflexibility. In this situation, it is argued that the evolution of international specifications such as IMS³ and SCORM⁴ are likely to reflect market power rather than educational needs.

The paper will review the main features of VLE and the principal approaches to their implementation and will analyse their contribution to learning with reference to two models "Cognitive Apprenticeship"⁵, and "Conversational Framework for the Learning Process"⁶

A range of approaches to the specification of user requirements will be discussed and conclusions proposed as to the implications for universities in Europe.

Although the process described above is constructed around general theoretical perspectives and policies, there is limited evidence that this process has been regarded as problematic from the perspective of the users, the students or participants in formal learning.

In the UK and other Anglophone countries, there is now a process of systematic review of the use and impact of Virtual Learning Environments [VLE]⁷. The benefits of this approach are cited as lower costs per student for Universities (economies of scale) as well as guaranteed high quality learning for students who wish to learn 'anywhere, anytime'⁸. The nature of the issues are well summarised by Paul Bailey of the UK Joint Information Systems Committee [JISC]⁹ who identifies the policy-driven nature of the use of VLEs as a means of increasing participation in Higher Education of people 18-30 to 50%, principally by progression from Further Education; Sue Timmis and Julian Cook¹⁰ address the issue of student motivation from an objectivist perspective concluding that VLEs need to provide learning opportunities not available elsewhere, clear expectations, guidance and support to enable students to respond to the potential of the VLE; Mark Stiles¹¹ identifies some debates about the learning process and advocates that we should be adventurous in our choice of pedagogies.

An Analysis of Users' Needs is a frequently used strategy for any major software implementation¹². The JISC PORTOLE¹³ project has developed a useful approach which involves a three-stage model:

1. "Identify possible objectives . The first step involved the collection of general set of potential objectives. These were identified from experts in the field of e-learning and resource discovery, (academics, librarians, and IT developers) and from the project team and collated by the project manager.

2. Decide on important objectives . Focus groups were conducted in order to discuss the importance of different potential features for the tool.

3. Select a final set of objectives which will lead to the evaluation analysis."¹⁴

This approach assumes that an organisation would take a corporate approach to the implementation of a VLE. In one UK University, the Senior Management identified a systems requirement in terms of creating links between such major Information Systems such as Student Information [Registration, Enrolment onto Modules, Recording assessment, and Publishing results], Learning Resources management, Access to the Campus through a single portal. A representative group of teaching staff was consulted on the implementation of the strategy and three VLE products were identified as meeting the system requirement. Three course teams were given limited support to pilot one of these products for a semester. At the end of the semester, a group of Bachelors' students conducted their final dissertation in order to evaluate the pilots. The reports were submitted to Senior Management who decided on the final product.

Mark Stiles has summarised an alternative view which identifies the importance of good course design, the need for a planned, pedagogical approach and the importance of staff development when selecting and implementing a VLE. ¹⁵The area of "appropriate pedagogy" has recently been widely debated and this issue will be returned to in the next Section.

A major commercial influence on these developments is the software platform WebCT [Web Course Tools] which currently dominates the UK market¹⁶ and is a major force in the rest of Europe¹⁷. "WebCT's vision is to work in concert with leading institutions to go beyond online course delivery and actually transform the educational experience. To that end, we will deliver state-of-the-art educational technology that supports a full range of teaching and learning styles and optimizes intellectual and technical resources."¹⁸

"Characteristics of successful online students - not all students are successful in online courses. Some students have trouble with time management, need the structure of a classroom, or miss the face-to-face interaction with other students and the instructor. The following are some characteristics students should have to be successful learning online.

• Discipline to complete projects by deadlines instead of waiting until the end of the semester.
• Motivations to read, write, and participate fully in class activities.
• Time to devote approximately 12 hours a week to a 3-credit course. [equivalent to 10 ECTS]
• Ability to work independently and in teams.
• Flexibility in dealing with technology problems.
• Self-starter does not procrastinate.
• Able to learn from the printed word. Majority of material and communication happens through reading and writing.
• Set aside specific times on a routine basis to 'participate' in the course.
• Can ask questions when they do not understand.
• Access to a current computer and the Internet.
• Good, basic computer skills.
• Possess time management skills."¹⁹

Let us reflect for a moment on ourselves and our students. Are these criteria fully realisable in practice? Do they take enough account of the competing pressures of everyday life?

An influential approach to teaching and learning in Higher Education has been the "Seven Principles for Good Practice in Undergraduate Education." This study suggests that "Good practice in undergraduate education:

1. Encourages contacts between students and faculty.

2. Develops reciprocity and cooperation among students.

3. Uses active learning techniques.

4. Gives prompt feedback.

5. Emphasizes time on task.

6. Communicates high expectations.

7. Respects diverse talents and ways of learning."²⁰

This framework was updated more recently recognising that the multi-functional nature of VLE implies the need to be more selective to ensure fitness for purpose²¹. For example, asynchronous staff-student and student-student communication by email (inside or outside a VLE) can improve understanding if all use it effectively (especially reading their Inbox regularly). However teachers' expectations of a regular pattern of "office hours may not be congruent with expectation of students of 'support at any time'".

The writer's experience with first year full-time students aged 18-20 is that their motivation tends to be instrumental and defined as much in terms of University as a social process and possibly a rite of passage. In particular, their level of competence and confidence in using ICT applications is often too low to enable engagement with a VLE. However, the minority of the students who did not have these handicaps were very positive about the possibilities of the technology in extending the range of their learning. The experience with part-time postgraduate professional students is quite the reverse and having built an initial peer group through face to face Induction at the beginning of a Semester, the group becomes more self-sustaining. This experience supports the Model that "Learning is built around learning communities and interaction, extending access beyond the bounds of time and space, but offering the promise of efficiency and widening access."²² The following view is fully supported: "A successful online learning community has many of the same characteristics as a 'real' community. It offers individual support to its members, so that they can feel safe to communicate openly, which in turn allows them to develop the shared vision that they need in order to learn together."²³

The principle and dominant model of e-learning in the Anglophone commercial world is 'blended learning'. "The term blended learning is used to describe a solution that combines several different delivery methods, such as collaboration software, Web-based courses, EPSS, and knowledge management practices. Blended learning also is used to describe learning that mixes various event-based activities, including face-to-face classrooms, live e-learning, and self-paced learning. Unfortunately, there's no single formula that guarantees learning, but here are some guidelines from NIIT on how to order your learning activities."²⁴

As indicated in the critiques referenced in the last endnote, there appears to be a range of responses to this clear approach. In the writer's experience with post graduate part time professionals, this concept often proves useful as an entry point to the conceptual "jungle".

"Almost anyone who has taught online would argue that the demands on online tutors are different from those on face-to-face tutors, although the general issues and situations with which they must deal are, in essence, the same. The online tutor must manage a course, guide students throughout the learning experience, motivate them, interact with them, assess them and deal with any conflicts or difficulties.

The differences in tutor role result from the characteristics identified above: the absence of non-verbal clues, the use of text as the main means of communications, and the constraints imposed by technology. However, any list of roles that can be produced should only be regarded as a general framework."²⁵

The COSE project at the University of Staffordshire²⁶, led by Mark Stiles, has made a major contribution to developments in the pedagogy of online learning.

"eLearning presents a challenge to the roles and responsibilities of academics and "support" professionals. Apart from the expected "quality processes" most traditional course development is done in an ill-formed and often quite ad-hoc way. ... Involvement of support professionals ... is very rarely integrated in any real way into the course design/development process.

eLearning, by its very nature, demands considerably more planning. However, there is little evidence that pedagogy is much considered in this process, with far too many staff seeking to model traditional practice onto e-delivery. Support, at all levels, is often either overlooked or not effectively used. Indeed, it is very often the case that developers have not thought through the reason why they are going to use eLearning in the first place. (Sometimes the reason for doing it is solely that funding exists!).

Overall, the essential amateurism of HE course design and development is thrown into sharp relief by eLearning."²⁷

An earlier paper put these issues in a sharp focus: "Whilst the ultimate potential of VLEs is unlikely to be realised until the standards and technologies are in place to facilitate their interoperation with each other and wider organisational and information systems, the view of the developers of COSE is that their successful deployment depends first and foremost on addressing the pedagogic issues associated with effective learning and ultimately on the overall quality of course design and learner support. The author contends that over-attention on the "features" provided by VLEs can lead to a "check-list" approach to VLE selection, which, coupled with inattention to the educational issues, can result in mere transposition of traditional teaching approaches to the computer, and result in a poor learning experience which is ineffective."²⁸

"There is a need for institutions to address within their curriculum the developmental needs of 'non-traditional learners' and most institutions have taken steps in this direction. Provision for this can be "separate" - where provision is specifically targeted at non-traditional learners or "integrated" where provision is aimed at developing requisite capabilities in all learners (Warren, 2002). Warren argues strongly for the integrated approach as the best means of meeting both the widening participation and "skills" agendas. This view is in concurrence with that of the author, who has previously argued that the development of the use of meaning-making, or semiotic, tools which are specific to given disciplines or professions and the reapplication of the tools and practices of the given culture, coupled with recognition of the relatedness of tacit and codified knowledge in the context of tasks or problems, is fundamental to the development of expertise (Stiles 2000a). This supports a view that skills are more likely to be 'transferable' when developed in an authentic context, rather than treated in a separate, "generic" way.

The author proposes that unless there is a national focus on the aspects of eLearning concerned with "form" (pedagogy and assessment) which is aimed at bringing these into balance with the current focuses on "content" (curriculum and resources) and issues of technology, students will be increasingly provided with isolating and passive learning experiences which, in turn, will impact most negatively on those very learners which the government is concerned to involve in FE/HE participation. This negative impact could well disillusion those staff who are currently becoming involved in the use of technology for the first time, and may result in a "backlash" among teaching staff and university managers, resulting in a set-back to the achievement of national goals and competitiveness. ²⁹"

These critiques and similar views on a crude transference of conventional approaches to the VLE context have identified a relationship between successful online learning and the use of constructivist approaches.

As indicated in the select bibliography, there is no shortage of recent work in this field. From a business perspective, it is generally agreed that "... in the US less than 25% of businesses evaluated the impact of training on business results. In the UK, we suspect that it is a lot lower." (Clark 2003:3) Apart from the projects cited below, the dominant approach is that of Donald Kirkpatrick 1959 four-level model³⁰. This puts forward the following structure:

The Institute of Employment Studies [IES] has recently published a study of Kirkpatrick's approach and approaches developed from the original model. It forms an excellent starting point for a review of approaches to the evaluation of training and learning as well as reviewing alternative models which focus more on the purpose of evaluation and on alternative measures. (Tamkin P et al, 2002) Main findings include the lack of an established causal relationship between ascending levels; little correlation between learner reactions and measures of learning or subsequent measures of changed behaviour (Levels 1 & 2); a failure to transfer training to the workplace (Level 3) which related to issues of how difficult the learner perceived the course (negative) and how much line managers were supportive (positive) (Level 4). (Tamkin P et al, 2002: xi-xiii)

Previous IES Research (Pollard and Hillage 2001) is cited which raises the key issue about e-Learning "How can you tell if it is working?" in view of the high costs of implementing the technology; the need to demonstrate the efficiency and effectiveness of this new learning solution; and the need to show organisational value of this (as yet) unproven process. The main approach identified was the measurement of performance, competences and intellectual capital. (Tamkin P et al, 2002: 18-22)

ASTD has conducted a major study on the evaluation of effectiveness and return on investment of e-learning and this makes clear that the availability of rich data within VLEs offers an important evaluation opportunity³¹.

A recent seminar paper concluded that:

"What is lacking is a theoretical basis and a coherent research framework. There is little systematic research into broad based issues and concepts, or the generation of transferable models and processes of evaluating e-learning or into the design of tools for analysing, rather than collecting, data. Furthermore, there are few papers written which collate the results of the existing research and classify it in an accessible way. Nor is there substantial evidence of work that extrapolates and tests generalisable principles arising from the case studies and surveys or which comments on the implications or application of these in a European VET arena." (Hughes and Attwell 2002/03)

The writer argues that with the increasing use of competence based national qualification frameworks³²; such highly functional approaches should form a significant point of departure for teaching and learning in Higher Education, especially in the light of the earlier discussion on "Blended Learning." In particular, there are significant implications for the working conditions of teachers, support staff, and learners of a move towards a "24/7" approach.

The bibliography of this Review indicates some important resources that are available for the evaluation of VLEs. The IHEP Benchmark study (Phipps & Merisotis 2000) has provided the basis for a pilot investigation of its appropriateness for a European cultural context on the JISCMAIL list "Evaluation of Online Learning" [www.jiscmail.ac.uk ]. A copy of this document will be published as an Appendix to this paper.

Another important resource is the "Evaluation Cookbook" (Harvey 1998) which makes effective use of a "recipe" approach to provide a range of practical approaches to the evaluation of learning technology (not just VLEs) for a wide variety of stakeholders.

At a more specialist level and building on the above work, the JISC Evaluation Toolkit (Conole 2000) provides a structured resource for effective evaluation. The writer has not been able to discover published evidence of its use outside the development context.

In summary, the state of the art in evaluation of the use of VLEs in Higher Education is a matter of "under-use" rather than under-theorising! As this review has shown, there is a wide variety of theoretical and practical approaches available for use.

In my view, it is unlikely that teaching and learning using VLEs in Higher Education will become more efficient and effective without significant applied research and evaluation.

John Konrad
John.konrad@dsl.pipex.com

A select bibliography of Educational Research on virtual learning environments [VLEs]

Britain, S & Liber, O, (1999), "A framework for pedagogical evaluation of virtual learning environments". A report presented to the Joint Information Systems Committee's Technology Applications Programme (JTAP). http://www.leeds.ac.uk/educol/documents/00001237.htm

Clark D (2003), "Evaluation of Learning", An Epic White Paper, Epic Group plc, Brighton UK [http://www.epic.co.uk ]

Connole G (2000), "An Evaluation Toolkit for Practitioners", Institute for Learning and Research Technology Bristol, www.ltss.bris.ac.uk/interact21/in21p06.htm

Cousin G, & Deepwell, F. (1998). "Virtual Focus Group Techniques in the Evaluation of an Electronic Learning Environment", In Oliver, M. (Ed.) ELT 98: Innovation in the Evaluation of Learning Technology Conference Proceedings, London: University of North London, 4-7,
 http://www.unl.ac.uk/tltc/elt/elt98.pdf 

Harvey J (ed.) (1998), "The Evaluation Cookbook", Learning Technology Dissemination Initiative, Edinburgh, www.icbl.hw.ac.uk/itdl

Hughes J & Attwell G (2002/03), "A Framework for the Evaluation of E-Learning", Paper presented to a seminar series on Exploring models and partnerships for eLearning in SMEs, held in Stirling, Scotland and Brussels, Belgium, in Nov 2002 and Feb 2003, http://www.theknownet.com/ICT_SMEs_seminars/papers.html

Joyes G (2000), "An evaluation model for supporting higher education lecturers in the integration of new learning technologies", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/joyes.html

McFadzean, E (2001a), "Supporting virtual learning groups: Part 1: a pedagogical perspective" Team Performance Management: 7, 3/4. 53-62.

McFadzean, E (2001b), "Supporting virtual learning groups: Part 2: an integrated approach" Team Performance Management, 7, 5, 77-93

Milligan, C (1998), "The role of virtual learning environments in the online delivery of staff development.

Phipps R & Merisotis J (2000), "Quality on the Line: Benchmarks for success in Internet-based Distance Education", The Institute for Higher Education Policy, Washington DC, April, www.ihep.com

Pollard E & Hillage J (2001), "Exploring e-Learning" IES Report 376, Brighton UK, www.employment-studies.co.uk

Report 1: Review of experiences of delivering TALiSMAN online courses". A report presented to the Joint Information Systems Committee's Technology Applications Programme (JTAP). http://www.leeds.ac.uk/educol/documents/00001230.htm

Oliver, M. (1999). "The ELT Toolkit", http://www.unl.ac.uk/tltc/elt/toolkit.pdf

Oliver M (2000), "An introduction to the Evaluation of Learning Technology", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/intro.html

Richardson, Julie A (2001), "Changes and challenges of academic lives through the introduction of virtual learning environments", Paper presented at SCUTREA, 31st Annual Conference, 3-5 July 2001, University of East London. http://www.leeds.ac.uk/educol/documents/00002517.htm

Richardson, Julie A & Turner A, (2000), "A Large-scale 'local' evaluation of students' learning experiences using virtual learning environments", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/richardson.html

Scanlon E, et al (2000), "Evaluating information and communication technologies for learning", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/scanlon.html

Taylor J, Woodman M, Sumner T, Blake C T, (2000), "Peering Through a Glass Darkly: Integrative evaluation of an on-line course", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/taylor.html

Tamkin P et al (2002), "Kirkpatrick and Beyond", IES Report 392, Brighton UK, www.employment-studies.co.uk

van der Veen J, & de Boer W, "W3LS: Evaluation framework for World Wide Web learning", Educational Technology & Society 3(4), http://ifets.ieee.org/periodical/vol_4_2000/veen.html

Notes:

1. A more general term is "online learning" sometimes more generally known as "e-learning". The latter term is used in a commercial context and also includes the use of such "off-line" material as CD-ROM and DVD. In French, the equivalent term is "l'apprentissage en ligne"- « Mode d'apprentissage basé sur l'utilisation des nouvelles technologies, permettant l'accès à des formations en ligne, interactives et parfois personnalisées, diffusées par l'intermédiaire d'Internet, d'un intranet ou autre média électronique, afin de développer les compétences, tout en rendant le processus d'apprentissage indépendant de l'heure et de l'endroit. » http://www.olf.gouv.qc.ca/ressources/bibliotheque/dictionnaires/Internet/fiches/8872865.html
2. Commission of the European Communities (2002), eEurope 2005: An information society for all, COM (2002) 263 final, Brussels, Page 13. http://europa.eu.int/information_society/eeurope/news_library/documents/eeurope2005/eeurope2005_en.pdf
3. The IMS Global Learning Consortium develops and promotes the adoption of open technical specifications for interoperable learning technology. Several IMS specifications have become worldwide de facto standards for delivering learning products and services. IMS specifications and related publications are made available to the public at no charge from www.imsglobal.org. No fee is required to implement the specifications. IMS is a worldwide non-profit organization that includes more than 50 Contributing Members and affiliates. These members come from every sector of the global e-learning community. They include hardware and software vendors, educational institutions, publishers, government agencies, systems integrators, multimedia content providers, and other consortia. The Consortium provides a neutral forum in which members with competing business interests and different decision-making criteria collaborate to satisfy real-world requirements for interoperability and re-use. (my emphasis) http://www.imsproject.org/aboutims.cfm
4. Sharable Content Object Reference Model [SCORM] provides a technical architecture for learning objects to be easily shared across multiple learning delivery environments. The SCORM specification describes an architecture built on specifications designed to address the US Department of Defence's (DOD) frustration with not being able to share distance learning courses among its different learning delivery systems.
   This set of specifications when applied to course content, produces small, reusable learning objects. These can be easily merged with other compliant elements to produce a highly modular repository of learning material. http://www.cetis.ac.uk/encyclopedia/entries/20011129121727/view /li>
5. "In traditional apprenticeship, the expert shows the apprentice how to do a task, watches as the apprentice practices portions of the task, and then turns over more and more responsibility until the apprentice is proficient enough to accomplish the task independently. That is the basic notion of apprenticeship: showing the apprentice how to do a task and helping the apprentice to do it. There are four important aspects of traditional apprenticeship: modelling, scaffolding, fading, and coaching. ... Our discussion of cognitive apprenticeship raises numerous pedagogical and theoretical issues that we believe are important to the design of learning environments generally. To facilitate consideration of these issues, we have developed a framework consisting of four dimensions that constitute any learning environment: content, method, sequence, and sociology. Relevant to each of these dimensions is a set of characteristics that we believe should be considered in constructing or evaluating learning environments. ... Cognitive apprenticeship is not a model of teaching that gives teachers a packaged formula for instruction. Instead, it is an instructional paradigm for teaching. Cognitive apprenticeship is not a relevant model for all aspects of teaching. It does not make sense to use it to teach the rules of conjugation in French or to teach the elements of the periodic table. If the targeted goal of learning is a rote task, cognitive apprenticeship is not an appropriate model of instruction. Cognitive apprenticeship is a useful instructional paradigm when a teacher needs to teach a fairly complex task to students." http://www.21learn.org/arch/articles/brown_seely.html
6. The main purpose of the Conversational Model for Knowledge Apprenticeship is to reveal the conversations taking place between an expert and a novice. Further, it is also to illustrate the building of capabilities in both a context-specific and in a generic sense.
   The CMKA is a model that builds upon two well-established teaching and learning theories: the conversational framework (Laurillard, 1993 Pg. 102-104) and cognitive apprenticeship (Collins, Brown, and Newman, 1989). Diana Laurillard's conversational model attempts to reveal the physical conversations taking place between an instructor and a student when involved in the teaching and learning arena.

16. See "WebCT, provider of the world's most flexible and widely used higher education e-learning solutions, announced today that leading UK higher education (HE) and further education (FE) institutions are rapidly choosing WebCT Campus Edition as their campus standard for e-learning. This announcement not only marks the growing demand for educational institutions to standardise to one Virtual Learning Environment (VLE), but also highlights WebCT's ongoing leadership in the UK HE and FE markets in delivering the most advanced educational technology available to support all teaching and learning styles. 'Enhancing the quality of education is vitally important and is something we are heavily committed to at the University of Edinburgh. High standards and changing student demands need to be met successfully,' commented Jeff Haywood, director, multimedia and learning technology services, The University of Edinburgh. 'Using a Virtual Learning Environment that conforms to international standards helps us to ensure that a continual level of high quality learning and teaching is experienced throughout the institution and that the overall management of courses is carried out in the most efficient and effective way.'" http://www.webct.com/service/ViewContent?contentID=17540607
17. WebCT is available in Dutch, Finnish, French, German, Spanish, UK English, Italian, Portuguese, Russian, Swedish. See http://www.webct.com/europe#1
18. WebCT's Vision: Transforming the Educational Experience http://www.webct.com/transform/viewpage?name=transform_webct_vision
19. California State University, Sacramento http://www.csus.edu/uccs/webct/student/character.htm
20. Chickering A W & Gamson Z F (1987), " Seven Principles for Good Practice in Undergraduate Education" AAHE Bulletin March http://www.aahebulletin.com/public/archive/sevenprinciples1987.asp
21. Chickering, A W & Ehrmann S C (1996), "Implementing the Seven Principles: Technology as Lever," AAHE Bulletin, October, pp.  3-6. available at http://www.tltgroup.org/programs/seven.html
22. "Applications and systems include:

This document was added to the Education-line database on 22 September 2003