JTAP Logo

A Framework for Pedagogical Evaluation of Virtual Learning Environments

Sandy Britain
Oleg Liber
University of Wales - Bangor

Contents:

1.4 Who uses VLEs?

In order to estimate the current usage of VLEs in UK Higher Education We conducted a survey of 100 UK Higher Education institutions using the instrument copied in Appendix 3 to determine the current usage of online systems for learning and teaching. The purpose of this survey was to discover which (if any) of the integrated VLE packages are currently being used and if so to obtain some feedback on their ease of use for performing various tasks. We were also interested to find out how many respondents reported using any on-line system for teaching and learning (e.g. putting lecture notes on the world wide web).

In the survey instrument we first asked respondents to describe any systems that are used for teaching and learning. In particular we asked for details of the use of virtual learning environments, conferencing systems, the world wide web and synchronous communications systems. The second part of the instrument queries respondents on the effectiveness of systems they use in a number of tasks. These were grouped according to the tasks that a VLE may be expected to assist with. The survey was then sent to the information services department of each institution.

The results of the survey were as follows:

Out of the 100 survey forms sent out eleven were returned by the cut off date.

Out of the eleven responses:

The number of survey forms returned was unfortunately too low to warrant detailed analysis of the results and any conclusions drawn from these results are necessarily speculative. Possible reasons for the low rate of return other than simple lack of willingness or time to participate are:

1.5 Why are VLEs relevant to Higher Education in the UK?

Despite the hype that inevitably surrounds anything on-line, 'virtual' or web-based at present, there are some clear and immediate benefits of these systems to students and teachers alike that concord with the recommendations of the Dearing Report (1997) regarding C&IT use.

Many advocates of VLEs within Higher Education are excited about their use because of their potential to allow a resource-based and student-centred approach to learning to be incorporated into their teaching, (e.g. Collis, 1996). One of the major blocks to adoption of this style of teaching and learning in universities is the extra time burden it places on tutors over the more traditional content-centred approach. Perhaps the key contribution that VLEs can offer is to allow a resource-based approach whilst also alleviating the tutor from an extra administrative burden.

There are various ways that VLEs might achieve this but as yet there is currently no established evaluation methodology from a pedagogical perspective with which to assess different systems. Most discussions and evaluative reviews of VLEs to date have tended to concentrate on the features, technical details and pricing of different systems. Whilst this information is invaluable in determining the choice of system, we argue in this report that an overriding concern is matching the pedagogical orientation of the system with its intended use in teaching and learning.

Mason (1998) suggests that there are three basic models of existing on-line courses

This provides a useful framework within which to consider the role of VLEs, and CAL systems in general, in UK Higher Education. Historically, and indeed until very recently, online courses and other CAL modules have consisted of either the content + support model or the wrap-around model. It is only with the development of fully integrated VLEs that implementing the integrated model on-line is a real possibility.

The first two models either replicate the structure of a traditional taught course or can be inserted into a course as a component without requiring any major change in approach from either staff, students or institutional infra-structural support. This represents a major difference with the integrated model.

Use of an integrated VLE implies that both staff and students are prepared to adopt an educational approach that is inherently more pro-active from the students perspective and that is in turn less pre-structured and more responsive to students requirements from the teachers perspective. Thus, first and foremost an evaluation framework needs to relate to the pedagogical process that the VLE is intended to support.

Aside from these educational factors there are administrative and management concerns associated with adoption of an integrated VLE, which cannot and should not be ignored. In order to maintain focus on the primary aims of this report, discussion of these factors is deferred until the end of the report.

1.6 Developing an evaluation strategy

As we have already mentioned, many comparative evaluations of software are carried out in a rather reductionist way, based on numbers of features, as opposed to looking at the function and usability of the overall system in the context of the human or organisational systems within which it is to be embedded. In the table in appendix 2 we provide a summary of the feature sets of all the systems outlined in this report. The asterisks that appear in the table denote a function that is implicitly available within the system even if it is not a feature. The information contained in this table is drawn from the manufacturer's or developer's own documentation, existing reviews and reports and personal experience of the systems. There currently exists a small number of comparative reviews of existing VLEs based on feature comparisons. Probably the most comprehensive of these is one conducted by Bruce Landon at the University of British Columbia. This is an on-line extensible review to which contributed data may be added. URLs for this and other online reviews are provided in the bibliography.

What becomes immediately apparent from an examination of the feature table in appendix 2 is that it does not distinguish a great deal between the different systems. The question is not as simple as deciding which system offers the most functionality for your money. Most of the systems can accomplish most of the tasks identified in the 'tools' column. However, this type of table says nothing about how easy or difficult it is to carry out a particular task. Moreover many of the systems are capable of implicitly supporting functionality by virtue of the web environment or the addition of third-party plug-ins. A good example is FirstClass. Although it is primarily a conferencing system, because of the web environment it is implicitly capable of providing an entire structured on-line course by authoring the appropriate web-pages and providing links to the discussions from them.

Most of the systems reviewed would be capable of supporting a content-driven model of online teaching and learning mentioned earlier (although in this case we might ask how much added-value they provide over simply using the web and e-mail?). What is less clear from the feature summary is how well different systems are suited to sub-serving the wrap-around or integrated models in the framework put forward by Mason (1998).

The answer, to our minds at least, lies not solely in the features of a system, but in how they are integrated to facilitate learning and administration and what metaphors are constructed to guide the way the system is used. In short we suggest that a more holistic approach to the evaluation of these packages is needed. Our aim is to ascertain the educational principles around which the system was designed and how well it fulfills our requirements as educators or learners.

The main thrust of this report then, is to propose an evaluation strategy from a pedagogical perspective. We have identified two key considerations for VLE evaluation strategies:

In order to address these issues, we consider two possible models upon which an evaluation strategy may be based. The first, and most obvious choice, is the Conversation Framework proposed by Laurillard, (1993) and one author Crawley (1999) has very recently published a paper in which an evaluation methodology for collaborative learning tools which is based on this model is proposed. The other is a model drawn from cybernetic approaches to modelling organisations known as the Viable Systems Model (Beer, 1981). The VSM provides a framework for us to think about the educational context into which a VLE is being placed and how the system helps a tutor manage the complexities of teaching a resource-based course with a large number of students.

2. Evaluation of VLEs using the Conversational Framework

In this section we explore the use of the Conversational Framework developed by Laurillard (1993) as an evaluation methodology for virtual learning environments.

We have already referred to the notion of a learning conversation (Laurillard, 1993; Dearing Report, 1997) as an alternative to a more traditional delivery approach to teaching. Whether VLEs are being used for distance learning or to enhance learning within institutions, it is our view that their most interesting role is as a medium for supporting constructivist and conversational approaches to learning. Thus Laurillard's conversational model offers itself as an obvious and interesting candidate for providing the basis of a pedagogical evaluation framework for VLEs.

The roots of the model lie in the Conversation Theory developed by Gordon Pask (1976), although Laurillard traces the need for dialogue in the learning process back to the Socratic method of philosophical enquiry. The centrality of dialogue in the model comes from the need for the teacher to unearth the student's mental constructs about a topic before negotiating the path to the target conception that is the goal of learning from the teacher's perspective.

The teaching strategy advocated in the model is based on the form of the interaction between teacher and student and not solely on the actions required of the student. The model advocates that action on the part of the student is constructed around the dialogue and should be supplemented by constructive and meaningful feedback from the teacher. Additionally there should be opportunities provided for student reflection.

2.1 The Conversational Framework

There are a number of key characteristics of the conversational model as applied to academic learning. These are drawn from Laurillard, (1993, pp.94-95).

Discursive

Adaptive

Interactive

Reflective

The conversational model as shown in the diagram of figure 2.1 depicts the workflow between tutor and student during learning. Certain activities (centre in blue boxes) are interactive and take place through some medium. Other activities (right and left in yellow boxes) are internal to either the student or the teacher. If we suppose that the medium involved is a VLE, then this model provides a clear set of requirements for evaluating the system's suitability for supporting the processes that form the basis of interactive learning.

 The Conversational Framework (adapted from Laurillard, 1993)

Figure 2.1 The Conversational Framework (adapted from Laurillard, 1993)

The primary workflow actions that take place through the interactive medium are as follows:

  1. Teacher presents / redescribes conception
  2. Student presents / redescribes conception
  3. Teacher sets up micro-world activities
  4. Student interacts with micro-world activities
  5. The system provides feedback on the action
  6. Student modifies actions in light of feedback

2.2 The conversational model and VLEs

This relatively simple framework offers much potential as a methodology for evaluating virtual learning environments. A similar proposal has been recently put forward by Crawley, 1999. In order to make this scheme work, all interactions must be assumed to take place via the medium of the VLE. It is then possible to make judgements about how well the VLE handles each of the individual interactions labelled in the model. The sort of questions one might ask are: How does the learning environment allow a teacher to present a conception? What does it mean for a teacher to set up a micro-world within a VLE? How can the student interact with the micro-world?

The model raises further questions about the mechanisms that support conversations. How easy is it to track conversations relating to a particular issue? Can conversations be enhanced by presentation of additional resources? Yet other questions relate to the overall flexibility of the system. How adaptable are micro-world structures once they are in use? How easy is to tailor them to individual students needs?

We might also want to ask whether the learning environment provides any extra tools to support reflection by the student on the relationship between the conversation at the level of descriptions and the activities they have completed or whether it provides structuring to assist the student in modifying their actions.

The interactions illustrated in the model do not necessarily occur in the linear order given above and the model involves recurrence at several levels. In particular the conversation may shift freely between the conceptual level and the level of actions. Any software environment supporting learning should allow this to occur naturally.

2.3 Evaluation criteria for VLEs

In order to evaluate a virtual learning environment using the conversational framework we need to establish what tools are provided within the software to allow dialogue and action to mutually influence each other to allow modification of both conceptions and actions on the part of the student as described above.

Additionally we need to be able to say what level of structuring is provided for each interaction. For example if e-mail is used as the tool for communication between tutor and student in a given VLE we need to know how the use of e-mail is embedded within the context of the dialogue about a specific topic or set of learning goals. Thus we might consider whether it is:

  1. simply a link to the users own e-mail client and existing mailbox
  2. a built-in e-mail client that allows messages to be stored and viewed from within the software
  3. a fully integrated client that automatically builds a conversation thread that is easily located from material relating to the specific topic of discourse.

Clearly b) and c) would be more sophisticated tools from the pedagogical perspective of the conversational framework than a).

Another issue that quickly becomes apparent is that the notion of a 'micro-world' takes on a different meaning in the case of VLEs than more traditional forms of courseware. In essence, the VLE provides the tools for a teacher to build a micro-world by allowing the teacher to construct learning activities enriched by multimedia resources and simulation programmes.

The crucial point from the perspective of the conversational framework is that the teacher should be able to construct the learning activity following a conversation with the student at the level of conceptions and the identification of a learning goal for the topic in question. Thus for any given learning environment we should consider how well the tools provide for both structuring conversations and actions and also how well they allow for integrating dialogue with actions.

Discursive Tools

All of the VLEs described in appendix A contain tools for conducting conversations. Naturally these rely to a great extent on e-mail and asynchronous discussion groups. It is important to consider how well the VLE leverages e-mail technology to support the conversation as an integral part of learning. For example, is the conversation accessible directly from the learning topic within the course structure or does the user have to move out of the course work in order to continue the conversation? Does the e-mail or conferencing tool allow attachments to be included with messages? And if so, can the attachment be extracted and embedded into the users course work structure? Does the tool allow learning goals to be specified and recorded on the basis of the conversation? Ideally the agreed learning goal should be in a prominent location with respect to the topic of learning.

Adaptability

How easy is it to adapt the activities associated with a learning topic according to the needs of an individual student or student sub-group as revealed by the conversation? This raises the question of whether students in the same group can be easily differentiated within the VLE once a course or learning activity is in progress.

Interactivity

A basic pre-requisite for a VLE is that it should be interactive. It is not enough for material to be presented to a student and then be tested on it. A VLE should allow the students to restructure the presented material, add resources of their own, annotate material, launch and run simulations etc. In other words the student should not merely be a passive observer of the 'micro-world' constructed by the teacher, but should be pro-active in shaping the 'world'.

Reflection

How does the VLE allow the teacher to help the student link detailed feedback on their actions to the topic goal? A concept-mapping tool might be a helpful feature in this respect. Alternatively, contextualised discourse for every level of the topic structure should be possible.

2.4 Constructing an Evaluation Framework

A suitable evaluation framework for VLEs using the conversational model could be constructed in a variety of ways. The first way that we suggest involves constructing a table that describes the tools that are available for each of the stages of interaction described in the model (see Table 3.1). In addition, it is helpful to also describe what support is provided for structuring the conversations and activities. This is an important element of the evaluation framework because it helps to describe how well integrated the tools are. Most of the tools that are incorporated into VLEs can be found separately in other packages, thus in some ways the very attraction of a VLE is the integration it provides.

Table 2.1 below illustrates the evaluation framework for VLEs using the interactions in the conversational model as criteria against which to identify the tools and level of structuring provided by the VLE. For each interaction we have provided examples of what we might look for in an integrated VLE.

 
Tools
Structuring
1. Teacher Presents Conception
What tools does the teacher have to hand:
Text, video, audio, images?
Can a teacher easily put together different multimedia formats for presentation of a conception? Can these be readily altered for re-presentation in a different way
2. Student Presents Conception
Can the student interact with the teacher through the system? Does the student have multimedia authoring capabilities? Even if text-only, how does the student communicate with the teacher? Clearly the dialogue between student and teacher is at the centre of the conversational model and how this is visually structured for both tutor and student is very important. Conversations should be at the centre of activity in the VLE rather than pushed to one side.
3. Teacher sets up micro world
Multimedia authoring tools for creating course materials, embedded or linkable simulation programs, testing software such as quiz creation programs etc. In a VLE the notion of micro-world can be applied at many different levels. The important point from the perspective of the conversational model is that it should be versatile enough to be adapted for an individual student on the basis of the ongoing conversational dialogue with that student.
4. Student interacts with micro world
See 3 above Again we can see this notion of micro-world at various levels. We are looking for more from the student side than simply being able to view content.
5. Tutor provides feedback to the student
Can the tutor use the communications tools to provide feedback to the student in the context of the students' activities? It might seem obvious that this would be true but the important point is that the feedback can be easily related to the action - i.e. any discussion thread should be linked to or embedded in the domain of actions.
6. Student modifies actions
Can the student return to the activities and modify their actions based on feedback received from the tutor?  

Table 2.1: An evaluation framework for VLEs using the conversational model

2.5 Testing the evaluation framework

We implemented this methodology for several of the VLE systems described in appendix 1. In most cases we only had access to demonstration versions of the software systems, on which to perform evaluation tests. Since many potential evaluators will also only have access to the demonstrations that are available, we felt this was a realistic test scenario. Evaluations for two of the example systems are illustrated below.

VLE1: WebCT

 
Tools
Structuring
1. Teacher Presents Conception
Primary presentation is through course material space (html pages). Other presentation of concepts is possible via e-mail, Bulletin Board and Whiteboard Learning Goals can be set for each page of content.
2. Student Presents Conception
Student Presentation Area permits uploading of student materials. Otherwise e-mail, BBS and Whiteboard  
3. Teacher sets up micro world
The micro world essentially is the course structure and materials. These are mostly prepared in advance. More fine grained materials can feasibly be inserted at this stage.
Any Web-interactive content can be incorporated
Dynamic tree representation allows hierarchical structuring of course 'path'. Each tree branch is a web page.
Calendar Tool allows time structuring.
4. Student interacts with micro world
  • Browsing content
  • Auto - Quizzes
  • Annotates content
  • Interacts with simulations and other live content
Study guide generation
5. Tutor provides feedback to the student
e-mail
BBS (full conferencing tool)
Whiteboard
 
6. Student modifies actions
Student can freely modify actions  

VLE2:Virtual-U

 
Tools
Structuring
1. Teacher Presents Conception
Course Space within the campus provides for initial presentation of ideas by the teacher. Subsequent presentation would occur in the Conferences Space or by private e-mail. Assignments can contain learning goals
2. Student Presents Conception
E-mail to the teacher or posting to a conference. Submission of assignments  
3. Teacher sets up micro world
The micro-world afforded by Virtual-U consists of many features: The course itself, the Gallery, the Library as well as assignments and activities associated with the topic. There are online help files to assist instructors with setting up different kinds of micro-worlds, assignments and projects with Virtual-U
4. Student interacts with micro world
The Campus metaphor provides a rich environment for student interaction with the system. Navigation between spaces is easy and flexible to use. There is a Glossary and extensive online help files to assist student working
5. Tutor provides feedback to the student
The tutor has access to a wide variety of information about student activities and progress. One-to-one feedback online would be given via e-mail or a conference of only two participants could be created.  
6. Student modifies actions
Student can freely modify actions  

One problem that we discovered when using the framework as described above for the tests was that several of the stages of interaction identified for the conversational framework require the same information regarding tools. It would be desirable to compress this information in some way. Also there is a semantic peculiarity that is thrown up by stages three and four - setting up and interacting with a micro-world. In the case of a VLE these stages really imply simply using the VLE. The important point is that the activities and structures set-up in the VLE by the teacher should be adaptable on the basis of prior discussion. This alone is a powerful discriminator of VLEs that are designed for delivery as opposed to those that are intended to be truly interactive

An alternative way of constructing the evaluation to take these factors into account would be to build the table using the principles rather than the stages of the conversation model: discursive, adaptable, interactive, reflective. We have implemented this strategy below for two further systems.

VLE1: TopClass

 
Tools
Structuring
Discursive
E-mail and discussion lists comprise the communications tools Both are accessible from the course homepage. Discussions can be integrated with course topics in TopClass server 3. Attachments can be included in e-mails to aid presentation of a conception.
Adaptable
Units of Learning material (ULMs) allow hierarchical structuring of course content. Very flexible structuring capability allowing individual assignment of learning content.
Date and time restrictions can be added to control access to course materials located in ULMs. This includes Tests
Interactive
The interactivity of TopClass is largely determined by the content that is placed in it and the way both course content and discussions are structured by the teacher There is no provision for students to add their own materials and resources into the course structure provided by the tutor.
Reflective
Students can view their own test scores. Feedback can be provided through e-mail and the discussion lists. There is no off-line working facility which would prove a restriction on students ability to access their work in the absence of a network connection  

VLE2: COSE

 
Tools
Structuring
Discursive
People in COSE are organised into groups which can be hierarchical. E-mail can be sent to the group from within the group manager tab. There is also a noticeboard for notices from group managers. There is no discussion group tool. COSE is organised around the sharing of work and has sophisticated tools for the creation and sharing of content, discursive tools however are limited. To compensate for this there is a great amount of flexibility in the creation and management of groups which means that conversations can easily be contextualised to the specific topic of interest. Unfortunately e-mails cannot be read and stored within COSE, so a conversational history cannot be built within the system.
Adaptable
Learning opportunities can be created easily and resources can swiftly be added by using the Gatherer, the Basket or the publishing tool. A learning opportunity can then be assigned to any group or subgroup The tools for re-use of content are excellent and once the tool has been in use for a short while individualised learning opportunities could be very quickly created and assigned. These tools are a major feature of the COSE design
Interactive
COSE is a highly interactive environment. The emphasis is on sharing work-in-progress with other members of a group. The facility for learners to create and manage their own groups and sub-groups means that the burden of work shifts from the tutor to the student as they become pro-active in shaping conversations and activities.
Reflective
By virtue of the organisation of the system around people rather than content, COSE allows feedback to be easily targeted at the right level and to be supplemented if necessary by additional resources drawn from the Basket. There is no off-line option with COSE.  

2.6 Summary

The examples provided above show how the conversational framework originally proposed by Laurillard might be adapted to serve as a way of evaluating VLEs. We have considered two methods of breaking down the elements of the model in order to assess the correspondence of a VLE to the requirements of the model, although undoubtedly others are possible.

In the first pair of examples we used the different stages of interaction as criteria against which to compare the system tools. This is useful because comparison of tools for stages 1 and 2 (teacher presents conception and student presents conception reveals the communication channels open to BOTH teacher and student and whether there are any special tools to assist with this process, such as a shared whiteboard for example.

The third and fourth stages deal with the level of actions in the conversational model. As we have stated, the 'micro-world' essentially is the VLE and it determines how much flexibility and scope there is for the teacher to set up and assign projects and activities within it. There are a wide range of approaches and some systems will be more suited to setting up the course structure in advance whilst others such as COSE are tailored towards setting up activities on the fly. The metaphor of the VLE has significant impact on how stimulating and easy to navigate the system is. WebCT, TopClass and several others adopt a simple strategy of linking tools to a course homepage. Virtual-U elaborates on this notion by providing a campus metaphor. Co-Mentor also builds on the notion of places to increase the contextual cues within the system. COSE has a detailed interface with a vista of the various projects and tabs denoting the various tasks such as gathering things and putting things in your basket. These metaphors not only affect the experience of using the system, but directly influence how teaching and learning is performed.

In the fourth stage, 'student interacts with micro-world' we considered how much scope there was for the student to shape their own view of the projects, to interact with the content and to add their own resources. Again systems vary quite considerably in this respect and this may be revealed here.

Stages 5 and 6 are less useful in the context of VLEs because there will, in essence, the same tools for a tutor to provide feedback as there were to engage in the initial conversation. Also due to the nature of a VLE in contrast to other types of educational software, the extent to which students are able to modify their actions is determined more by constraints laid down by the tutor than by the software.

Because we have used the conversational model somewhat differently in the context of VLEs, we formulated a second, simplified version of the evaluation table using the principles of the model as criteria. Although this version is more abstract in terms of the specifics of the model, it proved easier to use and draws out the crucial distinctions between systems at least as well as the previous version. We illustrated this approach using TopClass and COSE as sample systems. TopClass emerged as an all-round system that is more suited to a traditional style of on-line course. COSE, by contrast, is centred around collaboration in the sense of sharing content amongst groups of learners and the publication and re-use of materials. The conversational element in COSE has been pared down to a minimum.

One of the problems with the evaluation of software based on feature comparisons is that it is easy to lose sight of the wood for the trees. In this chapter we have presented the conversational model as a way of attempting to capture key aspects of a system from within a framework of educational 'good practice'.

However one of the main limitations of the conversational model is that it fails to reflect functionality associated with managing groups of learners. In the section we consider an alternative model with that aim at its core.

3. A Cybernetic Model for evaluating Virtual Learning Environments

3.1 Introduction

A key issue, perhaps THE key issue, for VLEs is concerned with how to allow for the management of groups of learners, for example when the VLE is being used to support traditional courses (ranging from 10-100+ students?). Formal teaching has a well evolved set of tools and approaches for dealing with groups of students en masse: these include lectures, seminars, tutorials and laboratory sessions. People are involved in the system in different roles - lecturers, postgraduate demonstrators, and personal tutors amongst others. All of these mechanisms and people are integrated and organised by a timetable, which constrains how and when they can take place; and how they fit into a larger scenario involving many courses and demands on space and time.

If VLEs are to be successful in enhancing educational quality, or if they are to achieve greater efficiency than traditional methods, they need to be able to address the types of issues that have previously been dealt with by the methods, techniques and tools outlined above. To be able to understand how they may attempt to do this requires some model that can generalise and explain these issues formally.

We have found Stafford Beer's work in management cybernetics extremely useful in suggesting such a formal approach, particularly his Viable Systems Model (VSM). This proposes that the structure of an organisation is primarily concerned with managing the complexity of interactions between its members, and with other stakeholders, in maintaining its identity and purpose(s) over time. The roles of organisational members define which communication channels are available to them, and with whom they can interact. In a highly hierarchical organisation (e.g. the army), the only valid communications are those with immediate superiors or immediate subordinates; in looser organisations (like universities) more communication options are available depending on committee structures and their membership.

 The organisation in its environment

Figure 3.1: The organisation in its environment

[NB. In figure 3.1, V stands for variety, the measure of complexity. In formal terms, it is the variety of the environment that is much larger than the variety of the organisation. Hence the organisation must amplify its variety or attenuate the environment's.]

The key problem for an organisation in maintaining its viability is how to manage the complexity of the situation it finds itself in. It is self evident that the world (or market) that any enterprise inhabits is much more complex than the enterprise itself, and yet somehow this has to be dealt with without the organisation being overwhelmed. How it does this determines the type of organisation it is.

It could put a lot of effort into understanding the environment, and try to respond to its perceived requirements (e.g. market research); or it could put a lot of effort into trying to shape the environment into something that suits the enterprises needs (e.g. advertising). Most likely it would need to adopt a range of approaches; but success is crucial. The most powerful but most devastating approach to managing complexity is to IGNORE it.

This elaboration also applies within an organisation: management needs to manage the complexity of the workforce or operations. The full picture becomes this:

 The Simple Viable System Model

Figure 3.2: The Simple Viable System Model

Again there are a range of approaches that have been adopted in organisations. One is to try to constrain this potential complexity by limiting the legitimate actions of the workforce through strict job definition and demarcation - the Taylorist model. The problem with this is that organisational flexibility is lost, and the abilities of the workforce are not fully exploited to the benefit of the enterprise. Another is to look for ways in which operations can become self-managing while remaining within the overall guidance of management. This requires a number of communication channels between management and operations for specific tasks. These include:

There also need to be communication channels that permit self-organisation and allow different operational elements to interact with each other without involving management. Self-organisation allows the operational elements to soak up much of their own complexity. How these channels are constructed, determines organisational flexibility and effectiveness. The schematic diagram in figure 3.3 indicates how self-organisation amongst sub-systems in an organisation can reduce the load on communication channels with the management without having to introduce a fixed hierarchical architecture.

 The Viable System Model: key operational communication channels

Figure 3.3: The Viable System Model: key operational communication channels

It is not the place here to go into much more depth about the VSM, we shall expand more on the relevance of figure 3.3 to education in the following section. However, there is one more aspect that will later be shown to be highly relevant to this report: that is the concept of recursion. The VSM is particularly powerful because it suggests that organisations can be seen as consisting of smaller BUT POTENTIALLY VIABLE organisations, working together to achieve mutual benefit. Each of these will also have to handle the same types of organisational issues, and so the VSM can be applied at multiple levels to understand and design how they function. The answers may be specific to each case - but the questions are the same.

3.2 The VSM and education

Education, (or at least formal education) depends on organisations. Some have formal and independent status (e.g. universities) others are more notional (e.g. the education system). Some last for may years, others come and go (e.g. courses). However, each is susceptible to interpretation using the VSM. Virtual Learning Environments represent both threats and opportunities for universities (and other educational establishments) - threats to traditional certainties, opportunities for increased effectiveness and efficiency. We need to understand VLEs in organisational terms to properly appreciate the types of impact they can have, and to be able to exploit them for our purposes.

We suggest that VLEs can be examined at 2 levels of recursion: the course level, and the institutional level. To date most VLEs have focused at the former, but increasingly they are being seen as providing institution wide solutions. However, the main focus of this report has to reflect the current state of the art, and so focuses at the level of the course. Before we proceed, we have to make an assertion that affects the forthcoming elaboration. We view learners as being the workers in the educational enterprise, and not as customers. Of course they have aspects of both, more so since they began to pay directly for access to courses at universities. But from our point of view, it is learners who have to work to create the change in themselves that we call learning. This process is facilitated, resourced, co-ordinated and monitored by teachers, but is undertaken by learners. So applying the VSM at this level we get this picture:

 the VSM applied to a course

Figure 3.4: the VSM applied to a course

The questions we need to address are how the organisational framework that enables the operation and adaptation of the courses does so.

Resource negotiation: how do learners negotiate their learning 'contracts' with their teacher? is this a one off or a continuous process? What are their mutual rights and responsibilities? What is the currency of this negotiation?

Coordination: can learners collaborate in creating their learning? How? What provision is there that can prevent exploitation?

Monitoring: how does a teacher monitor whether learning is happening, so that, if necessary, remedial action can be taken?

Individualization: how can each student find their own resources and advance their own learning independently of others? Can they contribute their discoveries to the group?

Self-organisation: what space or tools are available to let the learners organise themselves as a group, outside of the teacher's purview?

Adaptation: is it possible for the teacher to adapt the course and its resources in light of experiences gained during its operations?

These questions are equally valid for the organisation of formal teaching and for virtual learning: it is the answers that are different. Although there is considerable variation in non-technology based learning, the most common and traditional model can be characterised in the following way.

Resource Negotiation: there is usually no real negotiation - every learner gets the same resources (lectures, reading lists etc), although it may be possible to get some individual attention if the course group is not too large. Likewise the learner's responsibility is simply to return assignments or essays, or to pass examinations.

Coordination: in the most traditional of approaches, coordination simply does not exist - students are not expected to work together in any way, and if they do, no provision is made for this. Those teachers that do support group learning usually structure this within existing lessons, and no provision is made for group assignments or other actions outside the timetabled sessions. So the only coordination that applies across all formal teaching is achieved through timetabling, which ensures that all students get exactly the same diet of teaching and laboratory activities as everyone else.

Monitoring: this may be achieved through forms of continuous assessment, or through one to one tutorials, where teachers can examine how a student is developing their understanding and competencies. However, when class sizes are large, it is usually this aspect that suffers most. In some cases, teachers can amplify their monitoring capability by using post graduates as group tutors; in other cases personal tutors can play this role, although if the subject area is too far from their own expertise this can be less than satisfactory.

Autonomous learning: Students have always been able to use libraries to find additional materials to support their learning. This is perhaps what has ameliorated the lack of individual support offered by courses. However guidance on how best to find what they need has always been in short supply.

Self organisation: there is little evidence of any formal structures helping students to work organise themselves into work or study groups. While students may form study groups, neither time, space nor advice is provided for this, except in a few examples of study skills courses advising on group learning.

Adaptability: teachers can adapt their teaching to the needs of their learners, by changing the content of their lectures, the way they present them, and the supporting resources they use. However, these approaches tend to be coarse grained, responses to the perceived needs of the whole group, not individuals.

It is this last point that is the most significant. The formal set up implies that students following a course of study are to a large extent similar, and so an identical teaching approach will meet all their needs. Indeed, the process of selection also implies that the learner group is homogeneous. Formal teaching is designed to allow the teacher to treat the learner group as if they were one. However, the fact that the class does not consist of a single student constrains one key aspect - the ability of learners to engage in one to one conversations with the teacher. In a formal situation the conversational demands of a class of students would overwhelm the teacher, whereas if they had only one student, the balance between monologue (lecture) and dialogue would most likely be very different.

All these issues - resource negotiation, coordination, monitoring, adaptability, self organization, and individualisation can be affected by the use of technology. How they are affected depends on the facilities and flexibility particular systems offer. We believe that, in addition to the conversational criteria outlined earlier, these organisational criteria are crucial when considering the choice of VLEs.

3.3 Constructing an Evaluation Framework using the VSM

The methodology we shall use for evaluating the VLEs against the model just described is to outline what tools each provides for accomplishing the six tasks set out in the model. To illustrate this process we provided three examples of this approach using respectively: CoMentor, Asymetrix Librarian and Learning Landscapes.

VLE1: CoMentor

VSM Functionality
Support within VLE
Resource Negotiation The learning contract using CoMentor can be as loosely or tightly structured as a tutor wishes. A resource library is available within the system as are places for students to do their own or group work. CoMentor is designed to be used in institutions to supplement existing courses, rather than to provide a whole on-line course. CoMentor could be used to open the negotiation of a learning contract for the continuation of a course or to elicit understanding of a particular topic
Coordination Collaborative working is a key feature of Co-Mentor's design. Threaded discussion lists with sub-group membership allow students control over with whom they share their work. A shared concept mapping tool is also provided
Monitoring Tutors can monitor the discussions in CoMentor, taking part if appropriate.
Individualisation There is a repository of on-line resources within CoMentor which students are encourage to use to find their own materials. Additionally they have access to previous students work. This is a novel feature within a VLE and is potentially an extremely useful tool in helping to guide students thinking and to build on work that has already been accomplished. There is a list of people who can provide expert advice on selected topics.
Self-Organisation Students can organise their own work groups in CoMentor from which the tutor can excluded, if that is desired. In this sense CoMentor is a tool that belongs to the students as much as it does the teaching staff. Students can send e-mail to each other and tutors within the tool and there is a chat facility for synchronous conversations.
Adaptation The inherently dynamic nature of the CoMentor environment means that adaptation is not an issue in the same way as it is with systems designed to deliver courses. The content of CoMentor is encapsulated in the discussions, work and resources it contains and as such it is a continuously adapting structure.

VLE2: Librarian

VSM Functionality
Support within VLE
Resource Negotiation There is the capability for resource negotiation in Librarian. Students can self enroll in courses or modules which tutors have made available. The availability may be determined by previous conversations with the tutor or fulfilling conditions such as prerequisites. The student can browse available courses and also search the database for collaborations or lessons that meet specific criteria. Tutors can assign learning activities at various levels of granularity to individuals or groups
Coordination Administrators can set up collaborations to encourage students to work together. The software includes built-in threaded discussion lists but can also accommodate third-party collaboration software.
Monitoring Librarian has a powerful reporting facility that allows a tutor to monitor various aspects of a students performance and progress, including item analysis, content usage, organization membership and lesson score reports.
Individualisation Students can search the Librarian database for resources or other materials and of course the WWW. There is not an area for students to present their work or contributed materials to each other, although they can contribute materials as attachments to e-mails or discussion postings.
Self-Organisation Students cannot set up their own collaborations or construct their own learning activities in Librarian. Student interaction and organisation outside of the tutors influence is limited to individual e-mails
Adaptation The Librarian administrators interface supports easy assembly of an organisational structure using a hierarchical tree. Lessons can be organised into nested blocks and modules using this interface. This allows learning activities to be pulled out, modified and replaced at any point during a course. Because people are also represented in this structure, individual assignment of modified activities can easily be accomplished.

VLE3: Learning Landscapes

VSM Functionality
Support within VLE
Resource Negotiation In Learning Landscapes (LL) people can be assigned to learning activities or learning activities can be assigned to people (either individuals or sub-groups), thus the system can support either a traditional or resource-based approach. Conversations are central to the design of LL, so that as soon as a person is assigned to an activity, a conversation thread is started in the person's space within the activity. Further specialised resources or sub-activities can be negotiated from this conversation thread.
Coordination All members of an activity group or sub-group have access to a discussion forum accessible from the activity space. This organisation within the software naturally contextualises the discussions to the appropriate people and level. Students can also mail other group members individually in the same way.
Monitoring Tutors can monitor discussion threads and their own conversational histories with individuals. They can also check whether assignments have been returned and grades or view students' homepages for further information on individuals. More detailed progress tracking tools would be useful.
Individualisation The system includes a web browser in it. Students can use it to turn interesting on-line pages into resources in their personal resource structure. These can be posted to selected group members as sub-learning activities (there is no difference between the tutor's version and the student version).
Self-Organisation As mentioned above, students have the ability to construct and host learning activities without recourse to the tutor, thus a variety of special interest or project groups could be spawned within a learning activity group. This highly interactive approach requires the student to participate responsibly.
Adaptation LL is specifically designed for easy adaptation such that the initial course or overall learning activity sent out by the tutor is adapted and added to continuously as the course progresses.

3.4 Summary

In this section we have introduced the Viable Systems Model (VSM) and shown how the model can be adapted for use in an educational context (Liber, 1997). We then proposed that core processes within the adapted model could be used as criteria against which to evaluate the functionality of VLEs and also to gain a better understanding of the educational model or models to which they are best suited. One of the main benefits of the VSM is that it probes a whether a system can support a resource-based, student-centred teaching approach, amplifying the teacher's variety and attenuating the variety in the learner group to make the teaching approach viable - BUT without treating all learners as if they were all the same. We illustrated the VSM framework for evaluating VLEs using three examples drawn from the systems reviewed in section 2 - Librarian, CoMentor and Learning Landscapes.

One of the most important factors here is that students should have the capacity to organise themselves so that all the study is not solely tutor led. In terms of virtual learning environment this means that learners should be able to set up their own study groups and collaborations. Quite a few systems do not allow learners the permissions to do this. There should also be quite detailed information about different students interests available within the VLE. Thus students can locate others within their groups with whom they share common interests or goals. A good way to do this by leveraging web-technology is to have a link to each person's homepage as well as their e-mail.

Another important feature that emerges from this analysis is that students should be able to contribute their own resources and materials to the group (monitored by the tutor). Again this means that the responsibility for the shaping the learning content shifts from the tutor to the students. The tutor's time is then freed up to monitor discussions and engage in conversations or resource negotiation with individuals or sub-groups.

The third key feature of a VLE from this perspective is that it should be easy to continuously adapt and individualise the structures, content and activities it contains as the course progresses. It should also be possible to archive completed learning activities for future re-use.

4. Discussion and Conclusions

4.1 Discussion

At the start of this report we identified a number of ways in which virtual learning environments may assist with learning and teaching in higher education. Most of the benefits of VLEs lie in their potential to support styles of learning that are especially time-intensive for university teachers using traditional methods, but which have always formed a core part of a university education. We are referring, in particular, to:

Typically these learning styles have been supported using tutorials, seminars and small-group project work. However these activities have increasingly been squeezed out of undergraduate timetables in recent years due to factors such as increased time pressures on staff, increased student numbers and increased economic pressures.

As the admissions net has been widened, so too has the diversity of students - in terms of their backgrounds, age, experience, education and expectations. Yet due to the pressure of larger intakes, teachers have been forced to sacrifice precisely those activities that allow diversity to be managed positively, in favour of a uniform content delivery model that has economies of scale, but which essentially ignores the variety of the students.

As Laurillard (1993) states, "it is not that teachers want to teach this way any more than students want to struggle to learn in this manner, both parties are constrained to operate within a university system over which they have limited control and which is barely capable of withstanding external the pressures currently being exerted upon it".

The hope, expressed by university teachers and administrators then is that networked computer systems such as the VLEs we have reviewed here will allow more flexible teaching and learning styles to be adopted whilst at the same time reducing the time burden on the teacher.

We investigated a number of well-known or innovative VLE products (details may be found in appendix 1), some of which are commercially produced, others are the result of university development projects. For each VLE we compiled a feature set list from the various sources available to us (see appendix 2). We argued that although feature sets give an indication of the individual tasks a software package can perform they fail to capture the overall picture of how well designed the software is for supporting the integrated student activities listed above.

In response we put forward two models, one from an educational perspective the other from an organisational perspective that might be used to provide a more effective evaluation framework for VLEs. The two models are complementary in many ways, having different scope and orientation.

The conversation model focuses on interactions between an individual student and tutor. An evaluation from this perspective helps to identify whether a VLE is set up to allow individualised activities to be constructed for a student. The activity should be based on a prior discussion with the student that has identified any mismatch in the conceptual domain between tutor and student and the VLE should support this process.

The VSM model focuses on how the software helps a tutor manage conversations and the construction of individualised activities for a large number of students. In order for a VLE to effectively support the tutor in doing this, it must not only be easy to adapt on the fly but also provide for student self-organisation, resource-gathering and publication of material to the system.

In summary, used in conjunction, these two models provide an evaluation framework for VLEs. There is considerable scope for further elaboration of the scheme we have implemented here.

One of the problems with evaluating VLE packages that are currently available is that, with some exceptions, very few are currently being used for real within institutions. As a result, data and feedback from users is hard to collect. We expect this situation to change rapidly in the near future.

4.2 Conclusions

On-line learning technologies and virtual learning environments in particular are attracting considerable attention with in UK higher education and it is likely that emerging systems will have considerable impact on the way that learning and teaching is conducted in universities. Widespread uptake has yet to become apparent however, and the reasons for this are not entirely clear. Part of the problem that has been the primary focus of this report is that this field is still in an early stage of development and an appropriate framework for evaluation of different systems has not yet been established.

Perhaps the most important conclusion to be drawn from this report is that we are not providing an evaluation strategy that helps discriminate VLEs based on some notion of quality. Instead we have produced a framework that can help to reveal the underlying pedagogical assumptions and orientation of the software, thus providing teachers with a basis upon which to choose a VLE according to how they want to teach.

This point brings us back to the issue of whether choosing a VLE is an institutional-level decision or a responsibility that should be left in the hands of individual teachers. It raises the question of whether it is possible (or indeed desirable) to define teaching strategy at an institutional rather than individual level.

Higher education is changing. The student body is becoming less uniform and knowledge is growing and changing faster than ever before. Industry's demands are also more volatile, as they struggle to become more competitive in an increasingly globalised economy. Their employees need up to date knowledge and skills, and updating these requires training and education at the time and place they need it. While traditional courses will continue to be a major part of universities provision, the most likely growth area will be in 'lifelong learning' courses offered to industry or to individuals. The virtual learning environments of today and their successors are likely to form key strategical aspects of teaching and learning in universities of the future. The work contained in this report represents a first step towards recognising the educational and organisational requirements of this key learning technology.

References & Bibliography

Beer, S. (1981) The Brain of the Firm, 2nd Edn, Chichester: John Wiley.

Collis, B. (1996). Tele-learning in a Digital World. Thomson Computer Press, London.

Crawley, R.M.(1999) 'Evaluating CSCL - Theorists' & Users' Perspectives' http://www.bton.ac.uk/cscl/jtap/paper1.htm

Landon, B. online educational delivery applications: a web tool for comparative analysis http://www.ctt.bc.ca/landonline/index.html

Laurillard, D. (1993) Rethinking University Teaching - a framework for the effective use of educational technology, London: Routledge.

Liber, O. (1998) 'Structuring institutions to exploit learning technologies: a cybernetic model'. Alt-J 6 (1).

Mason, R. (1998) 'Models of Online Courses'. ALN Magazine 2 (2). http://www.aln.org/alnweb/magazine/vol2_issue2/Masonfinal.htm

Milligan, C. (1998) 'The role of VLEs in on-line delivery of staff development' JTAP Report 573. http://www.icbl.hw.ac.uk/jtap-573

National Committee of Inquiry into Higher Education: Higher Education in the Learning Society, 1997: http://www.leeds.ac.uk/ncihe/

Appendix 1

In the sections below we describe the features, technology and scenarios of use of a number of VLEs that are currently available. The information in the following descriptions is drawn from documents produced by the vendors / creators of the software, referenced reviews and evaluations and personal experience.

1. Learning Space

Overview

Learning Space is based on Lotus Notes and uses Notes Server technology to provide a secure environment with a rich set of tools. The basic architecture of Learning Space consists of five main databases that are used to manage different aspects of course development and use:

  1. The Schedule - provides an individualised schedule manager
  2. The Media Center - acts as a repository for multimedia course materials
  3. The CourseRoom - is a conferencing tool that allows the creation of threaded discussions for student interaction and group work on assignments
  4. The Profile Manager -contains basic information on participants similar to a homepage or online CV.
  5. The Assessment Manager - is a purely tutor's tool for privately testing, marking and giving feedback on participants work. It also allows a gradebook view of all participants on a course.

Learning space includes tools for browsing the web and inserting multimedia material into learning space documents. Links can be defined from Learning Space to multimedia content on the web. Asynchronous communications tools are based around e-mail, which is used for private one-to-one discussions and the CourseRoom where threaded private or public discussions can be facilitated by a tutor. Synchronous communications are supported by a variety of tools available with Learning Server 2.0 such as chat, whiteboard, video and teleconferencing. Additionally resources and other content may be exchanged via the Media Center. Students can maintain a progress-tracking folder containing a record of their own work and assignments throughout the course. Completed courses may be archived by the instructor for future use. A Portfolio is contained in every participant's Profile. This is a secure area for returned assignments and assessments which can only be viewed by the participant and the tutor.

Technology

Client-Server architecture. The Learning Server will reside on a Windows NT or Unix Platform. The client is cross-platform requiring either a notes-client (required for the tutor) or a web browser capable of supporting frames and an internet or LAN connection.

2. WebCT

Overview

According to the team that developed WebCT it is " a tool that facilitates the creation of sophisticated Web-based educational environments." (Goldberg and Salari, 1997). It can be used flexibly to create entire online courses, or to publish materials that supplement existing courses. All interaction with WebCT takes place through a web browser.. Essentially a webCT course consists of a series of linked HTML pages that define a path or "road-map" through the course material. The course content is supplemented by webCT tools which can be built into the course design by simply dragging the appropriate tool icon onto the web page. This creates an active link to a 'toolpage'.

There are three main aspects to webCT:

  1. A presentation tool that allows the designer to customise the look and feel of the course pages.
  2. A set of student tools that can optionally be integrated anywhere in the course.
  3. A set of administrative tools to aid the tutor manage the course when it is in progress.

A webCT course centres around a single course homepage, which appears whenever a student logs on to their course. This page contains links to the various course pages and the tools. The learner tools in webCT include: asynchronous communications tools (e-mail and conferencing) and also a chat facility; student evaluation and self-evaluation tools such as online quizzes and MCQs; an image repository; a glossary database; learner collaboration and presentation areas; content annotation, homepage generation, course navigation and searching tools. Tutors can track student progress including the number of times and when they have accessed the course.

Technology

This is a client-server system with all interaction through a web browser client. Supported server platforms are currently only Unix but an NT 4.0 version is to be announced.

3. TopClass

Overview

TopClass courses are constructed of Units of Learning Material (ULMs). These ULMs can consist of pages, exercises, or further ULMs themselves. ULMs can be freely exported and imported from course to course. In addition to course management, TopClass also manages student progress, tracking, and access to course materials.

The learner tools in TopClass include web browsing and embedding of hyperlinks into documents. Asynchronous communication and sharing is supported through e-mail, threaded discussion groups and BBS file exchange. Synchronous tools are not supported. There is a self testing facility and progress on coursework can be tracked by both tutor and student alike. It is also possible to produce individualised workplans for students.

The tutor tools in TopClass appear to have been well designed for ease of course construction and adaptation of courses in progress. ULMs are nestable learning units which are built into a course using a hierarchical outlining tool. Students are divided into groups according to a class model and access to discussion forums can be restricted to the class. Also ULMs can be easily created and assigned to particular subsets of students.

Technology

TopClass server software, web browser as client, LAN or Internet connection

4. Virtual University

Overview

Virtual-U like the other systems described is a server based integrated environment for education that uses a web client. One of the major differences between Virtual-U and other systems is the use of a campus metaphor within which locations and objects are used to represent the different tools and activities. The homepage contains a map of the Virtual_U campus which consists of :

Additionally, there is also a detailed help system providing course designers with pedagogical guidelines for the use of the system.

Technology

Web browser client, Server software runs on Unix or NT.

5. Web Course in a Box

Overview

Web Course in a Box (WCB) is one of the earliest integrated VLEs to have been commercially available. All access to a WCB course is through a web browser. It presents a customisable course homepage on log in which contains links to the various features of WCB. These consist of:

  1. A course syllabus providing an overview of the course, prerequisites, instructor etc.
  2. An announcements board.
  3. A course schedule. A tabular presentation gives dates and hyperlinks to the course units and materials themselves
  4. Student Information. This includes the students' homepage and e-mail contact.
  5. Learning Links. This where discussion forums, shared whiteboard and links to learning resources reside.
  6. Help / Utilities. This link takes the user to a panel for editing personal material and preferences and also provides a help system.

Technology

WCB server software and course authoring tools. Web browser client.

6. CourseInfo

Overview

CourseInfo is another example of a robust commercial VLE package which is based around a client-server architecture where the server software is accessed through a web client. It incorporates many features in common with systems described above which are accessible from a course homepage that is created and customised by the course instructor. Features include: An announcements board, course pages, course and staff information, an assignments space, communications tools, student tools and search tools. The communications tools include both asynchronous discussion facilities and chat and whiteboard functionality. Student tools include a calendar, homepage editing, a grade-checking function and a drop-box for sending assignments. There are also self-testing tools.

Technology

Blackboard Server for Unix or NT, web client.

7. FirstClass Collaborative Classroom

Overview

FirstClass is slightly different to many of the systems mentioned here in that it is primarily a conferencing system with robust e-mail and collaboration tools. But with the introduction of the First Class Intranet Server (FCIS) it also allows web publishing of course materials, personal websites, and rapid application development of applications to run either over the web or in the FirstClass client. First Class is probably the most widely used conferencing system in UK HE currently. A notable example is the OU who support over 35,000 users with FirstClass. Important features of FirstClass from a student perspective are the ability to work offline, search facilities, a 'chat' tool. From a teacher or administrators viewpoint the ability to publish web content easily means that on-line courses can be offered with FirstClass and interactive simulations can be built using the RAD tools. In addition the client UI is highly customisable.

Technology

FCIS server runs on MacOS or Windows NT and a FirstClass or web client required

8. Librarian

Overview

Librarian provides complete VLE functionality. Administrators have a well-designed interface for administration and course construction activities. A hierarchical tree model allows 'organisations' to be defined and placed in the tree. An organisation consists of members, learning activities and sub-organisations. This allows any organisational structure to be modelled in the tree. Learning activities may be similarly organised into blocks and modules, where each block may contain one or more modules and a module directly contains some learning content. Learning activities may consist of a combination of course materials with assignments and on-line discussions for collaborative work. Students may sign up for any block or module of interest, although pre-requisites on access may be set by the course administrator. Librarian also provides progress tracking and search facilities.

Technology

Librarian is a Java-based learning management system that runs on a Windows NT or UNIX platform. Its client/server architecture enables learning delivery across the Internet or intranets and is designed to scale with the growth of an organization.

9. ARIADNE

Overview

ARIADNE is a the result of a European funded project form a consortium of European universities with its primary aim to enhance sharing and re-use of computer-based teaching material. In the words of the project leaders "ARIADNE explicitly addresses four aspects of the teaching process: producing computer-based teaching material, managing this material to permit sharing and re-use, assembling courses, and delivering courses to students. ...ARIADNE has developed tools for creating simulations, multiple choice questionnaires, self-assessment exercises and tools for segmenting text and video. Hypertext generation is also supported with the aid of a text conceptual segmenting tool."

Technology

ARIADNE server, web client (Java enabled)

10. CoMentor

Overview

CoMentor is a web-based software system, developed at the university of Huddersfield, that is primarily focussed on facilitating discussions and collaborative work on-line. At the heart of the system is a MOO architecture implemented on a web server with spaces in which students can participate in real-time or asynchronous discussion. In addition there are learning tools to support and structure debates and collaboration. In particular students can work together over examples from archives of previous students' work. The system includes an individual work-space, a group work-space, and a resource archive. There is also an announcements board. CoMentor is not intended to provide complete on-line courses but rather to enhance existing courses.

Technology

CoMentor runs on a Unix platform and can be accessed from any Java-enabled web client.

11. CoSE

Overview

COSE is a VLE that arose from a project at the University of Staffordshire to build tools for the creation of study environments rather than delivery of materials. The software was designed to support a constructivist pedagogy based on principles of good practice. The idea behind COSE is that a course is a group of people to whom learning opportunities can be flexibly assigned, rather than the more traditional view that it is a body of content to which people are assigned. Cose has built-in asynchronous discussion tools and work-sharing tools. COSE provides a publication mechanism for content that includes metadata and is one of the few systems described here to take the issue of metadata seriously. Tutors can easily assign and de-assign learning opportunities to groups, sub-groups and group managers can be created and all subgroups have their own noticeboard. Learners can create content as 'COSE' pages and optionally attach other sorts of files to them. Thus submission and return of assignments is made simple.

Technology

Like the other systems COSE is platform independent using a web client. The COSE server software should run on any server platform running Apache 1.0+ and Perl 5.003+. It has been tested on NT4, Linux 2.0, and DEC Unix.

12. Learning Landscapes

Overview

Learning Landscapes is a Java software application developed at the University of Wales - Bangor that supports on-line interaction between student and tutor to allow the negotiation, creation and management of study programmes. One key difference between Learning Landscapes and other systems described above is that is a distributed rather than client server application, this means that no server management is required and that offline working is feasible. The software is specifically designed to handle the management of resource-based collaborative learning. With Learning Landscapes you can:

Technology

Java 1.2 application, no server required.

 

FCC

WebCT

TopClass

LSpace

VirtualU

WCB

CourseInfo

Librarian

COSE

CoMENT

LL

Ariadne

Teacher Tools

Resource Management Tools

                       

creating /importing content

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Store resources

*

yes

*

*

yes

yes

yes

yes

yes

yes

yes

yes

Add metadata

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Add description

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Add/play multimedia content

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

People Management Tools

                       

Store & view learner data

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Add / remove learners

 

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Track learner activities

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

Course Management Tools

                       

Course structuring

*

yes

yes

yes

yes

yes

yes

yes

yes

*

yes

*

adding resources

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

creating assignments

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

performing assessments

*

yes

yes

yes

yes

 

yes

yes

yes

yes

yes

yes

rapid course revising

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

*

create discussion groups

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

*

Student Tools

Resource Management Tools

                       

Web browsing

No

*

*

*

yes

yes

yes

yes

yes

yes

yes

yes

creating / importing content

yes

yes

 

yes

yes

yes

yes

yes

yes

yes

yes

yes

Store bookmarks

*

yes

yes

yes

yes

yes

yes

 

yes

yes

yes

yes

Add metadata

*

       

yes

   

yes

yes

yes

 

Add description

*

       

yes

   

yes

yes

yes

yes

Play multimedia

*

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

yes

People Management Tools

                       

view people data

*

 

yes

yes

 

yes

yes

yes

yes

yes

yes

yes

Homepage authoring

yes

yes

yes

   

yes

yes

 

yes

yes

yes

?

Learning Management Tools

                       

Calendar tool

*

yes

 

yes

yes

 

yes

       

yes

Self-testing tools

*

yes

yes

yes

   

yes

       

yes

searchable resource archive

*

yes

 

yes

   

yes

yes

yes

yes

 

yes

create discussion groups

yes

               

yes

yes

?

                         
 

FCC

WebCT

TopClass

Lspace

VirtualU

WCB

CourseInfo

Librarian

COSE

CoMENT

LL

Ariadne

Interaction Tools

E-mail

yes

yes

yes

yes

yes

yes

yes

yes

*

yes

yes

yes

Noticeboard

 

yes

yes

yes

yes

yes

yes

 

yes

yes

   

File exchange

yes

yes

yes

yes

yes

 

yes

 

yes

yes

yes

yes

asynchronous discussions

yes

yes

yes

yes

yes

yes

yes

yes

*

yes

yes

yes

Chat

yes

yes

 

yes

yes

 

yes

   

yes

   

Whiteboard

*

yes

 

yes

 

yes

           

VideoConferencing

*

   

yes

               

Virtual Learning Environments Questionnaire

Part 1. Respondents' Details

Notes: This section may be left blank if you wish to remain anonymous.

Name of Institution....................................................................................................

Respondent's Position ................................................................................................

Contact e-mail:.............................................................

Part 2. About teaching and learning at your institution

Approximately how many students are enrolled on degree courses at your institution? ................

What proportion of your students are enrolled on part-time courses?......................(approx.)

What proportion of your students are distance learners?......................(approx.)

Part 3. Description of systems currently in use

Notes: In this section we are trying to gather information about what systems (if any) are currently used at your institution

Do you currently use an integrated Virtual Learning Environment for teaching and learning at your institution? YES / NO

If YES, please mark as appropriate:

WebCT TopClass LearningSpace Virtual-U

CourseInfo Web-in-a-Box Forums FirstClass Classrooms

Other (please specify) .......................................................................................

...........................................................................................................................

Do you use a conferencing system or other groupware product for teaching and learning purposes? YES / NO

If YES, please mark as appropriate:

Domino / Notes FirstClass MS Exchange Collabra / Newsgroups

Other (please specify) .......................................................................................

...........................................................................................................................

Do you use the WWW for teaching and learning? YES / NO

If YES, please mark as appropriate:

Lecture Notes

Other resources (Please describe below)

Other activities (Please describe below)

...........................................................................................................................

...........................................................................................................................

...........................................................................................................................

...........................................................................................................................

Do you use videoconferencing or other synchronous communications system to support group working, either at a distance or with computer-based resources? YES / NO

If YES, please mark which technologies you use:

VideoConferencing Whiteboard Shared Browsing Chat

Please also mark which (if any) of the following products you use:

CU SEE ME NetMeeting Timbuktu Netscape Conference

Internet Relay Chat

Other (please specify) .......................................................................................

...........................................................................................................................

Do you use net-based virtual laboratories, interactive simulations or other interactive courseware? YES / NO

If YES, Please supply details.......................................................................................

...........................................................................................................................

...........................................................................................................................

Do you use any other system or systems that you think we may be interested in? YES / NO

If YES, Please supply details.......................................................................................

...........................................................................................................................

...........................................................................................................................

If you have answered no to all of the above questions, please return the form to us at the address at the top - this information is still valuable! If you have responded affirmatively to any of the above questions please fill in the next section to the best of your ability.

4. The properties of virtual learning environments

Notes: In this section we are aiming to query your impressions of the system that you are currently using in order to gain a rough idea of the capability of various systems. Unless you are using an integrated virtual learning environment not all of the subsections will be relevant. These sections use a Likert scale rating scheme based on the suitability of the tool for performing various tasks. The scale key is as follows:

1 = not at all

2 = poor

3 = satisfactory

4 = good

5 = excellent

For each question please circle the appropriate number.

4.1 structuring of courses, resources and learning content

Does your system provide a facility for tutors to construct courses by embedding content, task descriptions, on-line resources into a structured course outline? YES/NO

If Yes, please rate your system on the following criteria.

How suitable is it for aggregating learning units and / or content?

1 2 3 4 5

How adaptable are the structures once they have been created and are in use?

1 2 3 4 5

Can a structured problem space be easily created for students to work in?

1 2 3 4 5

How easy is it to support individual learners' needs?

1 2 3 4 5

How specifically can informational resources or references to resource locations be incorporated into projects? (e.g. hypermedia support)

1 2 3 4 5

Is the system suitable for incorporating a wide range of multi-media resources and content?

1 2 3 4 5

How easily can non-proprietary software components (e.g. JavaBeans / ActiveX components) be embedded in the tool

1 2 3 4 5

4.2 Handling Workflow

A fully integrated VLE should be able to handle all the workflow requirements between students and tutors in a flexible and context-sensitive way. It should also be able to handle a range of pedagogical models from a traditional delivery model to collaborative approaches.

Does the tool provide shared workspaces? If so please rate the quality

1 2 3 4 5

How well does the VLE provide for negotiation and development of individual learning plans?

1 2 3 4 5

How easy is it for tutor and student to collaborate in constructing a problem domain within the virtual environment?

1 2 3 4 5

How well does the system support collaborative working of a number of students on the same project?

1 2 3 4 5

Does the system support submission of assignments from student to tutor? If so please rate the quality

1 2 3 4 5

Does the system support recording and return of assessments to students? How well does this facility meet your needs?

1 2 3 4 5

How well does the system accommodate off-line as well as on-line working?

1 2 3 4 5

4.3 Communication Tools

If the system uses e-mail to support one-to-one conversations, how easy is it to use in a teaching / learning context?

1 2 3 4 5

If conferencing software is integrated with the system in order to support group discussions and group working, how satisfactory is it?

1 2 3 4 5

If the system uses any form of synchronous communications, how useful are these components?

1 2 3 4 5

4.4 Student Profiling / Progress Tracking Tools

How well does the tool allow tutors to track students' progress?

1 2 3 4 5

How rich a picture of an individual student's background, interests and aspirations does the tool provide?

1 2 3 4 5

4.5 Integration Issues

How well integrated are the various tools provided by the system?

1 2 3 4 5

How easily can content from external sources be integrated into the environment?

1 2 3 4 5

Is the system IMS compatible?

1 2 3 4 5

How well does the system inter-operate with administration systems (e.g. student records databases etc.)?

1 2 3 4 5

4.6 Functional / Technical Issues

How easy is it to set up and maintain the system?

1 2 3 4 5

How easy is it to learn to use the system for tutors and students?

1 2 3 4 5

How time-consuming is it to enter data into the system?

1 2 3 4 5

This document was added to the Education-line database on 25 November 1999