Evaluating the quality of eLearning resources
Paul Riddy and Karen Fill
Centre for Learning and Teaching, University of Southampton
Paper presented at the British Educational Research Association Annual Conference, Heriot-Watt University, Edinburgh,
11-13 September 2003
Faced with a bewildering array of eLearning materials, modules, courses and programmes how can tutors or learners decide which ones will best meet their needs and be of sufficiently high quality? Which of a range of standards should eLearning developers adopt to understand and address the requirements of teachers and learners and ensure the quality of their offerings?
A European team from universities in England, Spain and Germany, and training organisations in Italy and Greece, has worked together over two years to produce a methodology for the evaluation of on-line open and distance learning materials and Internet based programmes of study. The team members were multi-disciplinary and represented both academic and commercial backgrounds. Their expertise spanned course design, development, production and delivery. Some members had skills in technological implementation and course management. The aims of this EU funded project, Methodology for the Analysis of Quality of Open and Distance Learning delivered via the Internet, (MECA-ODL1) were to:
provide guidance for improving the overall quality of e-Learning courseware
facilitate buying decisions for tutors, learning portal managers and learners support e-Learning developers and suppliers in the quest for quality.
The MECA-ODL evaluation methodology essentially looks at characteristics, which define the quality of a learning resource in all stages of its development and implementation, in order to assess if it is likely to be sustainable and deliver quality learning for users.
The project developed through four main stages:
- Compilation of a compendium of existing ODL resources from within each partner country, and non-partner countries, and review of the scope of ODL materials
- Development of an evaluation methodology, based on the review results and existing evaluation schema
- Development and testing of the on-line evaluation tool
- External evaluation of the on-line tool.
The major outputs from the two-year project were a methodological guide for the analysis of quality in eLearning and a software tool that can be used on-line to evaluate materials, modules, courses and programmes.
This paper presents an overview of some evaluation schemes which existed at the project's inception, explains the MECA-ODL methodology and contrasts it with these, discusses the results of external evaluation, and makes recommendations for further development of the methodology and tool.
During the project there have been significant developments in the specification of Learning Objects (LO's), defined as "any entity, digital or non-digital, which can be used, re-used or referenced during technology supported learning" (IEEE, 2001). LO's are likely to play an important role in the construction of eLearning resources, and readers are referred to Littlejohn (2003) for an extensive discussion of this subject. It is envisaged that evaluation methodologies will need to take account of this in the future, but LO's are not explicitly referenced in the current MECA-ODL evaluation methodology.
Existing Standards and Specifications
The main body involved with standards on a global scale is the International Organisation for Standardisation (ISO). This represents a hybrid of organisations from the public and private sector, with members from 147 countries, and its standards are truly international: "International Standards provide a reference framework, or a common technological language, between suppliers and their customers - which facilitates trade and the transfer of technology." (ISO, 2003). The ISO standards, which relate to this topic, are:
Similar in concept, the Total Quality Management (TQM) model is widely used, but is not actually a standard. This represents a structured system for satisfying internal and external customers and suppliers by integrating the business environment, continuous improvement, and breakthroughs with development, improvement, and maintenance cycles while changing organizational culture. (IQD, 2001)
These, and similar schemes, are concerned with broad aspects of quality. They are complex, weighty and, for many educational developers, probably unusable. They enshrine notions of quality that start with development and continue beyond production, but are actually of little use for education.
Other bodies, working in the area of Learning Objects (LO's), have now produced tightly defined specifications and standards which consider the technical implementation, interaction (interoperability), and educational qualities of LO's. Two of the major international players in these specifications and standards are the IMS Global Learning Consortium (www.imsglobal.org) and the Institute of Electrical and Electronics Engineers (www.ieee.org). Activities specific to the UK are coordinated by the Centre for Educational and Interoperability Standards (CETIS, 2003). These developments are already having an impact on the development of eLearning and future evaluation tools.
Evaluation of eLearning Resources
To date a number of evaluation schemes for the quality of eLearning resources have been developed. Unlike MECA-ODL, other examples given below do not cover all aspects of resource design and implementation. The differences are outlined in each case and summarised in Table 1 below.
E-Learning Courseware Certification
Promoted by the American Society for Training & Development (ASTD), E-Learning Courseware Certification (ECC) is
for asynchronous Web-based and multimedia courses (and) recognizes courses that excel in usability and instructional design. (...) The ASTD Certification Standards Committee, composed of e-learning experts, academicians, instructional systems design practitioners, and other learning leaders in the industry, has created these standards (which) are supported by examples, clarifications, definitions, scoring criteria, and other supporting information. (ASTD, 2002a)
The ASTD scheme evaluates compatibility, interface, production quality and instructional design of e-learning courseware and now offers an on-line pre-evaluation tool:
which allows the end-user to pre-screen an asynchronous learning course against the 19 standards that are in place. The browser-based tool allows the end-user to open a course in one window and the tool in another so that the course can be reviewed. The tool keeps score based on input from the end-user. (ASTD, 2002b)
Final certification is still done by external evaluators. ASTD is planning to enable access to an online database of certified courses.
Differences: The scope of the evaluation criteria does not engage with the readiness of an organisation to deliver eLearning, including the existing training culture. The criteria potentially offer a good assessment of other aspects of a course, but it is difficult to tell from the information available. ASTD offer a certification service for eLearning courseware, for which they charge a fee, and presumably have experts well versed in all aspects of course design and development to ensure comprehensive evaluation.
Quality on the Line
The American Institute for Higher Education Policy reported on '24 benchmarks considered essential to ensuring excellence in Internet based distance learning' (IHEP, 2000). These were arrived at after a review of the literature and carrying out research with institutions delivering distance learning to determine which benchmarks were in use. The benchmarks are divided into seven categories of quality measures, namely:
Institutional support Course development Teaching/learning process Course structure Student support Faculty support Evaluation and assessment.
Differences: This is a manageable set of criteria by which to evaluate the quality of a course but, in our view, does not provide a complete enough picture, especially in the areas of course design, content and production.
Consumer Based Quality Guidelines for Learning Technologies and Distance Education
This consumer guide is based on extensive research into the literature related to technology assisted (distance) learning. The quality of education and training is defined by what makes distance learning modules effective and efficient from a consumer's perspective. The criteria are presented as a serious of 15 questions, the scope of which is illustrated below:
- Acquired content skills and knowledge should be:
Relevant, transferable, specific....
- Necessary learning skills are acquired for:
Course completion, lifelong learning, self-directed learning
- Completion takes the form of credit or credentials that are:
Recognised by professional accreditation bodies and other educational institutions, equivalent whenever learnt on-site or at a distance, transferable...
- Return on investment of the learner's time, finances and energy meets expectations for:
Accessibility, objective benefits, effectiveness, efficiency, customer satisfaction.
Differences: As consumer guidelines these provide a good starting point for a potential learner to assess if a course is likely to provide what they are looking for, and indirectly suggest evaluation of the design, content and implementation. However, the consumer focus leads to the guidelines being output based, and they do not offer sufficient detail for a complete evaluation.
The basis for evolving the evaluation methodology was the compendium of eLearning resources, the standards and evaluation methodologies outlined above, and the experience of the team. The methodology developed encompasses seven phases, spanning requirements ranging from commercial to Higher Education (HE):
Conception - considers the readiness of the organisation to engage with eLearning Analysis - establishes eLearning needs and methodologies Design - pedagogical design and development process Content - implementation of design including selection of technologies Production - building the application, content, development team, sequencing Delivery - marketing, course team, management, evaluation, certification Evaluation - course review, and feedback/development cycle.
Each phase has a fully defined set of criteria giving a total number of 140 criteria. An example from the Conception phase is given in Figure 1 below.
Three categories of potential users of the guide and tool were identified:
Developers of eLearning modules / courses Users of eLearning modules / courses (learners) Resellers of eLearning courses (trainers / tutors).
After developing a range of criteria, the project team considered each to minimise duplication and finalise their allocation to each category. Each criterion was also weighted, on a scale of 1 to 5, to denote its relative importance in an overall evaluation. These are recorded in the paper guide (see Figure 1).
Phase: I. - Conception Criteria
I.1 Building e-learning strategy.
To build an e-learning strategy means to take into consideration and change not only issues of technology but also issues of learning effectiveness.
The learning culture in the organisation has been considered.
The impact of learning culture on effectiveness should be considered.
The learning background and interests of the organisations' staff have been considered.
The impact of staff factors should be considered.
Figure 2: Example from the printed guide of the Conception Phase
Note: the D/ U / R columns are used to indicate which categories of user the criteria are applicable to: Developer, User or Reseller.
Comparison of eLearning Evaluation schemes
A summary comparison of the schemes discussed above with the MECA-ODL scheme is given in Table 1 below. The table indicates the comprehensive scope of the MECA-ODL criteria compared to the other evaluation schemes. The descriptions above also highlighted the differences in terminology, and in the selection and organisation of evaluation criteria into categories. For example criteria under QoL course development category would fall into several of the MECA-ODL categories. The CBQG are all framed as evaluative questions, but are equivalent to some of the MECA-ODL criteria.
Table 1: Comparison of eLearning Evaluation schemes
Notes: ECC - E-Learning Courseware Certification
QoL - Quality on the Line
CBQG - Consumer Based Quality Guidelines (for Learning Technologies & Distance Education)
Ö - indicates the scheme has one or more evaluation criteria within the corresponding MECA ODL category
The other assessment schemes do not provide an indication of the importance of the criteria (the weightings), or recommend different sets of criteria for developers, users and re-sellers. These become particularly useful in the on-line implementation of evaluation methodology, the evaluation tool
The MECA-ODL online tool
The online tool is currently available in English, German, Greek, Italian and Spanish at http://wipaed-dbase.sowi.uni-bamberg.de/eLearning/WebObjects/mecaODLtool. At registration a user selects the appropriate category - developer, reseller or user - and is presented with the relevant phases and criteria when evaluating an eLearning module or course. The evaluation process involves awarding a 'score', again on a scale of 1 to 5, for the implementation of criteria (see Figure 2).
Figure 2: Data entry screen for application evaluation
An evaluator may choose to ignore or score each criterion presented. Evaluations can be partially completed and returned to at a later date. The tool calculates an overall score for each phase, displaying this as a number. Scores for all the criteria are also shown as numbers and graphically (bar charts) and are automatically compared to the MECA-ODL 'ideal' that has been derived from the weightings agreed by the project team. Examples are shown in Figures 3 and 4
Figure 3: Numerical comparison of application with the ideal
The tool was designed with flexibility for users to input their own set of weights for each of the criteria, but it is important that the same set is used if cross-comparisons between applications are required. At administrator level, the base set of criteria and reference weightings can be also be changed. The criteria, general descriptors, and operational instructions are stored in a database, making it straightforward to create a version in another language.
Figure 4: Graphical Comparison of Quality
As mentioned earlier, the tool was designed to span the spectrum of eLearning evaluation ranging from commercial to Higher Education (HE). In reality, within HE the full range of criteria will be rarely used, as many of the criteria in the early phases are not considered, or in some cases relevant, in the HE course selection process. This is brought out in the example below.
Evaluation of PosFix
PosFix was developed at the University of Southampton in 1993, as a small, standalone application for learning about position fixing, with an emphasis on electromagnetic systems. It was designed specifically for delivery in an on-line environment using a 'resource base' approach (Hall, 1994), and was implemented within the Microcosm environment (Davies, 1994). The application was still in use by undergraduate and MSc students in 2000.
PosFix was designed for undergraduates, postgraduates, and newcomers to Marine Science, with content suitable for beginners to intermediate users. It replaced sections of taught courses at undergraduate and masters level, aiming to provide basic knowledge that would be used in hands-on practical sessions at sea. Development was funded by the institutional TLTP project. The primary considerations were infrastructure readiness for delivery, educational design including integration with the existing courses, formative and summative evaluation. The application was designed to be introduced via a tutorial, and its use was seen as contributing to the transferable skills of the learners.
The evaluation using the MECA-ODL tool was undertaken from the perspective of a developer, when the maximum number of criteria can be scored. PosFix was used in a blended learning mode, and developed for an academic teaching environment, so many of the criteria relating to distance learning delivery and commercial training were inappropriate. Most of these were in the Delivery, Analysis and Conception phases. An example of the graphical output for the Delivery Phase is shown in Figure 5. Most of the scores for criteria used were between 60% and 80%, with the weaker areas being:
- Conformance to usability guidelines - these were not well established when PosFix was designed
- User control of interactivity
- User ability to record notes
- Availability of user tracking information.
The last three were largely a function of the development environment.
Figure 5: Graphical Comparison of Quality for the Delivery Phase of PosFix
Note: missing columns illustrate the large number of criteria that could not be scored for PosFix.
Overall PosFix scored highly. Principal weaknesses were in the design and content which were not adaptable for different learning styles, and non-conformance to metadata standards that are becoming increasingly important in today's development climate.
The evaluation highlights the difference between an academic and commercial environment. Within the University we were working with a largely pre-defined audience. Some commercial training companies specialise in tailoring training packaging to meet and the needs of a small group of learners.
The MECA-ODL guide and tool have been assessed by several external evaluators, in HE and commercial training, in each of the countries involved in the project. All agreed an evaluation tool was needed and common views of the benefits of the MECA-ODL approach were deemed to be:
it provides a useful checklist for building an application it can be used as a guide for developing other training/educational applications (e.g. digital training manuals) the graphical comparison function is very useful for comparing different applications the concept of weighted criteria the flexibility to adapt for own needs.
They also made many perceptive observations and suggestions for further developments that the team hope to address if further funding is made available. These include:
- Language and documentation
o more user friendly
o clarify language of guide
o more information on pedagogic models in guide
- Provide 'case study' example of use
- Distinctive sets of criteria for different purposes, e.g. commercial, academic
- On-line training course for evaluators
- Wizards - e.g. to set up an evaluation (criteria set)
- Repository of example on-line evaluations
- Link with learning objects metadata.
Although some of the original assessors commented unfavourably on the large number of criteria, recent discussions with interested parties who tried out the tool at the University of Southampton suggested there are not too many. Users of the guide and/or the tool should retain the flexibility to use them all or select their own sub-sets.
Planned developments of the Tool
Evaluation of eLearning resources requires a broad skill set. Users without any background in eLearning development and implementation would find it difficult to carry out an evaluation. Making a lot more information available via the user guide and on-line help system would go some way towards alleviating this problem. Ideally would-be-users would complete a module on eLearning evaluation to support use of the tool.
The main developments in the methodology are in removing some ambiguities in the criteria, and producing revised sets of criteria for different user groups. In conjunction with this it would be useful to have a simple way to set up the criteria and weightings for a particular user perspective. It is anticipated this could be done through developing 'wizards' which select the criteria set according to data input by the user. For example, one such wizard could provide a much-reduced criteria set for the novice eLearning evaluator.
This paper describes the development of an evaluation methodology for eLearning resources, the implementation of this as an online tool, its appraisal to date, and points out directions for future development. The results suggest that such a tool will be an invaluable aid for designers and developers of eLearning materials, who can use the criteria set as a checklist. It could also be very useful for users and resellers of materials, who want to compare the characteristics of different applications. However, our use has shown that complete evaluation of an application requires access to information which is frequently not available, or hard to find. For such methodologies to become a useful 'standard' for evaluation, it is essential that application developers are primed, and prepared, to make the information available. As with any pedagogic evaluation, significant background in learning and teaching methods is required to check if the pedagogic aspirations of an application have been implemented effectively. This requires skilled individuals, who are also comfortable with eLearning technology, to carry out the more comprehensive evaluations. Finally, it would be important to ensure consistency between evaluators if a meaningful database of evaluations was to be constructed.
ASTD (2002a) The ASTD Institute E-learning Courseware Certification Standards. Available online: http://www.astd.org/ecertification/standards.htm (accessed 04/07/03)
ASTD (2002b) Pre-Screening Tool Helps With E-Learning Courseware Certification. Available online: http://www.astd.org/ecertification/self_assess.htm (accessed 04/07/03)
CETIS (2003) What is CETIS? Available on-line: http://www.cetis.ac.uk/static/about.html (accessed 15/07/03)
Davies, HC and Hall, W and Heath, I (1994) Microcosm: an Architecture for Integrating Third Party Applications. In Proceedings of the ECHT'94 Workshop on Open Hypermedia Systems, Edinburgh.
Hall, W and White, SA and Hutchings, GA (1994) The Resource-Based Approach to Teaching & Learning. In Proceedings of the TLTP/CTI Conference on Evaluation, Dissemination and Implementation, Cardiff.
IEEE (2001) Learning Object Metadata Working Group, Scope & Purpose. Available online: http://ltsc.ieee.org/wg12/s_p.html (accessed 04/09/03)
IHEP (2000) Quality On The Line: Benchmarks for Success in Internet-Based Distance Education. Available online: http://www.ihep.com/Pubs/PDF/Quality.pdf (accessed 04/07/03)
IQD (2001) Definition of Total Quality Management. Available online: http://www.iqd.com/hoshin_def.htm (accessed 04/07/03)
IMS (2003) IMS Learning Design Best Practice and Implementation Guide.
Available online: http://www.imsglobal.org/learningdesign/ldv1p0/imsld_bestv1p0.html#1502383 (accessed 15/07/03)
ISO (2003) What 'international standardization' means. Available online:
http://www.iso.ch/iso/en/aboutiso/introduction/index.html#two (accessed 06/08/03)
ISO 9000 (2000) Selection and Use of the ISO 9000:2000 family of standards.
http://www.iso.ch/iso/en/iso9000-14000/iso9000/selection_use/selection_use.html (accessed 04/07/03)
ISO 14915-1 (2002) Software ergonomics for multimedia interfaces - Part 1: Design principles and framework. Available online: http://www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=25578 (accessed 04/07/03)
Littlejohn, A. (ed) (2003) Reusing Online Resources. London: Kogan Page.