Developing competent e-learners: the role of assessment.
Open University in Scotland, Edinburgh, UK
Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria, 28-30 August 2002
ABSTRACT We know assessment plays a major formative role in driving student learning appropriately, but what implications does this have for online courses? Is it more important than in a face to face context, or less so? Should we reconceptualize the ways in which we assess students, or are existing methods, tried and tested in conventional teaching and learning situations, appropriate? This paper draws on recent research of student and staff perspectives on two networked courses at the Open University, which employ constructivist pedagogies. It discusses the practical implications of implementing these online pedagogies and illustrates the powerful formative effects, both intended and unintentional, of assessment on student learning and behaviour on these courses.
The use of electronic media for flexible course delivery provides a new context for teaching and learning. The emphasis on asynchronicity brought by email and online conferencing brings with it an increased scope for flexibility in study routines, and meets a growing demand for part time study, continuous professional development and lifelong learning.
Electronic media offer a variety of ways of presenting or structuring learning opportunities which were not previously available on distance courses. They offer new ways to access and combine information, and the possibility to keep in touch on a more regular and continuous basis, so that students need no longer work in isolation, but belong to an electronic "community of learners" (Collis, 1998).
E-learning has recently achieved prominence as the "Philosopher's Stone" for future development in Higher and Further Education, and yet there is little common understanding as to its meaning.
Mason (2002) outlines the wide spectrum of interests and understandings attached to the term, from those with social constructivist approaches, whose aim is to make use of the communicative potential of online learning, to those who have a more behaviourist approach, and see it as a convenient medium for content delivery and testing students. There is also a wide variation in the types of student who may be involved in e-learning, in terms of age or motivation, their situation, which may be campus based, or remote, or a combination of both, and also the discipline which they are studying.
As part of this debate, a recent ESRC manifesto describes networked e-learning as:
"..those learning situations and contexts which, through the use of ICT, allow learners to be connected with other people, and with shared information rich resources. Networked e-learning also views learners as contributing to the development of these learning resources and information of various kinds and types."
This paper concentrates on those courses where a constructivist approach is adopted, by using online media to support distributed collaborative interaction and dialogue, and sometimes access to information rich resources. Such courses place importance on understanding, rather than on memorising and reproducing facts, and on the contribution of social interaction and collaboration to learning.
Constructivist philosophy accommodates a family of closely related pedagogies, which optimise the potential of networked environments. The family includes collaborative learning (McConnell, 2001), activity-based learning (Macdonald & Twining, in press), resource-based learning (Macdonald, Heap & Mason, 2001) and problem-based learning (Ronteltap & Eurelings, 2002).
These four pedagogies lay different emphases on particular facets of constructivist philosophy, as their names suggest. They all operate by providing opportunities for students to learn by engaging in activities, which might involve collaborative work, or problem solving, or open access to electronic resources. In practice, many networked courses adopt all four approaches to varying degrees, alongside more conventional modes of delivery.
The pioneers in e-learning have tended to concentrate their studies on postgraduate courses, often having small numbers of students, where e-learning has been the subject of study, as well as being the medium used for interaction. Many useful lessons have been learnt from these courses, however, e-learning is rapidly being mainstreamed at all levels of study, for a wide variety of courses.
At the OU UK there are now 275 courses involving 100,000 students in which online media are used for tuition. Over 16,000 FirstClass conferences have been set up for course related tuition, as well as student run peer support through the student union. In addition, by 2005 all students will be encouraged and expected to use web based systems for administrative services and non-academic student support; this will no doubt raise the general level of computer literacy, and also student expectations of online support.
What is a competent e-learner?
Courses using electronic networks make certain demands on the students who study them. Inevitably, using a networked computer as a study tool involves basic ICT skills which need to be mastered, and this eventually leads to a familiarity with the relevant hardware and software tools.
Beyond this, students need to learn how to use the computer as a study tool. They may need to accommodate screen based study, with its limitations on flexibility, and the implications for time management; in addition to learning how to manage files effectively. Approaches to writing and sharing information in electronic form offer enhanced possibilities for the re-drafting of scripts, as well as the use of information from a variety of sources. This presupposes some understanding of plagiarism, and the role of academic evidence in developing an argument. Arguably, all these skills and adaptations, once learnt, are fairly readily used in a variety of situations.
Courses which are networked will by definition offer students greater access to the opinions of peers and the resources of the web, and students will need to develop a self-directed approach study, and adopt a critical approach to these resources. Of course, under traditional university models of teaching and learning, encouraging students to develop any independence and self-direction in learning is a lengthy and gradual process (Perry, 1970), which develops throughout a graduate course of study. Since networked courses with constructivist approaches have been introduced at all levels of the undergraduate curriculum, e-learners may be exposed to unfamiliar demands at an early stage in their academic career.
Self-directed study using electronic networks requires competence in two major areas: information literacy and online collaborative learning, in line with the need to cope with collaborative interaction and access to information rich resources. The term information literacy has been in use particularly in the US and Australia, in connection with self-directed lifelong learning (see for example Bruce & Candy, 2000). Its characteristics include recognising the need for information, being able to identify and locate it, gaining access to it, then evaluating, organising it and using it effectively (American Association of College and Research Libraries, 1999). Students need to recognise the gaps in their knowledge, in order to establish what they need to find out, and Hill (1997) underlines the significance of subject knowledge, in providing the framework for further exploration and research, in addition to confidence and metacognition. It is clearly not enough to teach students searching techniques, and then to assume that they will be competent investigators. Indeed the support of students in this area is highly problematic, and McDowell (2002) describes a wide spectrum of approaches to the use of electronic resources within courses.
There is similar complexity in online collaborative learning. Salmon (2000) proposes a number of progressive stages of development, which include access and motivation, socialisation, information exchange, knowledge construction and development. These stages illustrate the interplay between competence, and affective factors such as growing confidence, motivation and group dynamics. And of course if students are to communicate effectively within an academic discipline, then they need to become familiar with the language of a discipline and the academic genre. Lea & Street (1998) maintain that this familiarity with the discourse is a defining factor in students' abilities to read and write appropriately within a discipline. In fact, this familiarity grows as they practice writing conference messages on course topics, and reading, or eventually responding to messages from others.
Finally, if students are required to collaborate to undertake a common task, as opposed to making optional contributions to a conference, then they need to practise team working and negotiation skills, group decision making and task management (see for example Schrage, 1990). Again, affective issues may be significant here, for example group cohesion, and the evolution of mutual trust. It follows that the whole process of competence in online collaboration certainly requires practice, and will take some time to develop.
In short, I have argued that the competent e-learner will have developed communicative and interpretive ability using electronic media. This involves the use of computers as an effective study tool, but also implies the development of critical and analytical abilities to work as a self-directed learner, together with the communicative abilities to work with, and learn from peers. In other words, competence in e-learning has many parallels with competence in learning, and is probably acquired as a developmental progression, alongside developing confidence and metacognition, and a growing understanding of a discipline. However, the use of online media for study, and the ready availability of information rich resources on the web, means that students may need to develop competence and self direction at an earlier stage in their undergraduate career than they would in more conventional distance courses. Course design and outcomes will need to reflect and support these needs.
A significant body of research supports the view that the design of assessment is critical in determining the direction of student effort, and that the formative value of assessment is vital in providing a channel of communication between students and their mentors (see for example Black & Wiliam, 1998). This role for formative assessment is increasingly important for campus based universities, as well as in a traditional distance learning context (Higgins, Hartley & Skelton, 2002). But to what extent can the formative power of assessment be deployed to develop competent e-learners, and does it differ in any way from formative assessment to develop competent learners?
In this paper I discuss the lessons learnt on assessment design for developing competent e-learners, by drawing on two qualitative studies of two networked courses at the UK Open University; both studies are reported in detail elsewhere (Macdonald, Heap & Mason, 2001; Macdonald & Twining, in press). Both courses engage students in learning overtly about e-learning, but I believe that the lessons learnt here will be applicable to any networked course which makes use of activities such as those described above, because it is likely to make similar demands on its learners.
The courses described here were both 60 point 2nd level undergraduate courses. In line with common practice at the OU UK, they were designed for part-time study, and each presentation ran for six months. Both courses adopted a constructivist approach and made use of the networked environment to engage students in a variety of activities. They were both assessed by a series of assignments spaced throughout the presentation together with a final examinable component.
The Technology course THD204 : "IT and Society" explored social and technological issues associated with IT developments. It was presented from 1995 until 2001, to approximately 1500 undergraduates each year. The course adopted a resource based and collaborative approach, delivered through a combination of printed course books, a CD-ROM indexed library of academic articles, and an internet Home Page which gave access to resources for further reading.
Empirical data is drawn from a case study of perspectives on the assessment of this course, which was undertaken over a three year period with three cohorts of students and their tutors. Data were gathered from 21 initial in-depth interviews and observations. These were followed by a series of computer conferences to which 800 students and 50 tutors were joined over two years; they ran throughout each course presentation, and provided a running commentary on the course. They were supplemented by 100 telephone interviews (Macdonald, 1999).
The Education course E211: "Learning matters: challenges of the information age", was presented in 2001 to 200 students. The course aimed to empower students to deepen their understanding of learning, and offered a variety of perspectives on learning, with particular reference to the growing influence of ICTs. It placed an overt emphasis on experiential learning and reflective analysis, whilst being relatively low in print delivered content, in comparison with conventionally produced distance courses. Course activities represented an integration of three types of task, in addition to course readings and videos:
- Participation in online conferencing and internet searching with fellow students;
- Hands-on experience of ICT as a tool for structuring reflection on learning;
- Maintenance of a portfolio in which they recorded personal learning experiences and made notes on course readings and various other activities.
A case study of both students and tutors on their perceptions of the course assessment was carried out at three points of enquiry, immediately after three of the assignments, using course conferences and targeted email questionnaires. A total of 10 out 12 tutors, and 20 students contributed feedback.
Creating e-learning opportunities at critical points
If a course is based on active learning and experiential approaches, then it is critical that students actually do the activities they are set, so that they engage with the learning process and derive benefit from the course. In a classroom situation there is some expectation that students, once in the room, will participate, at least to some extent, in the activities set them. Distance learners are not the same.
Intuitively one might expect little guarantee that students will undertake any element of a course in the prescribed manner, unless there is sufficient motivation to do so, and Lockwood (1992) describes the balance of "costs and benefits" undertaken by distance learners when considering whether to undertake in-text activities. Whether e-learners are any less likely to participate than other distance learners is difficult to say. Some students might be attracted to the "fun" element in activities, others might be motivated by online interaction, whilst others might see online activities as intrinsically "lightweight" in comparison with traditional course content. Obviously, this issue of participation is of critical importance where activities form a central part of the course. It is even more critical where activities involve online collaboration, because non-participation by one student impacts on other students.
Our own experiences illustrate this point. Throughout the course E211, students were expected to engage in online discussion in groups of 15-20, which were moderated by their tutor. The course materials provided them with discussion points to follow through in their tutor groups. Mid-way through the course, students were given an online collaborative activity, on which they were expected to work in sub-groups of four to six students. They were required to search for resources on the Internet and to reflect on their experiences of online collaborative working.
It appeared that for about half the tutor groups, participation had waned since the start of the course, and the sub-groups had achieved limited success in the collaborative task. Arguably the extent of success was related to a variety of factors, and the skills and moderation tactics of the various tutors undoubtably played a significant part in this (Mason & Bacsich, 1998). However, feedback from both students and tutors revealed that the greatest perceived hurdle to successful participation was the lack of integration of assessment with the collaborative task. At the same time, an impending assignment which was not related to the activity, materially detracted from student efforts, as these comments describe:
"Assessment tasks take priority and however valuable the learning opportunity might appear to be, it will be sacrificed in the name of survival."
"..timing and incentive are both important - a number of my students commented that they would have completed this activity (and taken a more active part) but stopped when they felt they had to give priority to work on assignments."
"..an impending assignment reduces participation significantly, and the very short gap between [assignments] 3 and 4 has more or less killed activity in my conference stone dead...Some of my students who began hopefully in the spirit of the course have been put off conferencing simply because they began to feel they were providing useful material for others who read but did not contribute."
There will always be students who recognise the value of online participation for their learning, and appreciate the benefits of belonging to an online community. But the only time when most students will undertake activities is when they are linked to assessment. The assignment will always take priority, and may detract from non-assessed activities. Of course, there are always exceptions, and some students will ignore activities even when they are linked to assessment (Thomas & Carswell, 2000).
An assignment which requires reflection on a topic of online debate, or on the application of a particular skill in operating course software, helps to ensure participation by most students, if it is timed to coincide with the relevant activity. The following is an example, which illustrates how an assignment can cover course content while also encouraging students to undertake an online collaborative activity:
Assessment may not necessarily be effective in measuring the learning which takes place, and the practice of awarding marks for participation, or the nature and form of contributions can be problematic, because of the need for reliability if several markers are involved. For example, Goodfellow (2001) describes how online assessment practices were considerably more subjective and difficult to monitor than conventional marking, because of uneven levels of participation and a requirement for flexibility in the assessment of those who were readers rather than contributors. In fact, Knight (2002) points out that summative assessment of 'uncontroversial' course content will always be easier to achieve than any assignment which sets out to measure the process of learning.
Nevertheless, assessment is invaluable as a way of affording students the opportunity to learn at critical points in the course, and a few marks can be very effective in providing that opportunity. In this respect it is certainly more significant for online courses than it is for conventional campus based courses.
Formative feedback and the developing e-learner
I have argued that the road to competence in e-learning is a complex one, in which the student must acquire competence in basic transferable skills before they are equipped to undertake tasks demanding of higher order literacies. Because of this complexity, and the degree to which much competence in e-learning is embedded within an understanding and familiarity with a discipline, it becomes important that the process of study be overtly supported in course outcomes, whatever the course. Further, the development of the more complex competences such as information literacy in electronic environments builds upon a familiarity with basic skills, therefore students may need incremental support throughout the course, to develop these skills gradually.
Our studies have illustrated how the assessment strategy can be designed to provide this support. Of course, the extent to which this is important will depend on the competence of the students at the start of the course. And perhaps five years into the future this will be less significant that it is at present.
For example, early evaluation work on THD204 showed that many students experienced serious problems with information overload when working on some resource-based project work, in spite of the provision of a computer based tutorial designed to help them learn how to use the indexed CD-ROM library. The resource-based activity was too complex for many students to handle, because not only did they need to be competent in negotiating the environment, they also had to formulate queries effectively and to assess the relevance of the material retrieved.
An exercise in an early assignment required students to practise the basic skills and become competent with the environment.
Our evaluation showed that this was a partially effective strategy, in that the students were better prepared for coping with resource-based study in subsequent project work. However, even by the end of the course many students remained unconfident with the resource-based approach and still felt that they were lacking in information literacy. This probably reflects the longer timescale needed to develop such competence, and also the relationship of information literacy to an understanding of the discipline, which for second level students was as yet poorly developed.
My second example is taken from E211, where an assignment required students to construct a hypertext essay using a hypermedia authoring tool called Hypernote. In this essay, they discussed course concepts, using hypertext to illustrate the relationships between them, and links to video clips as illustrative examples.
When asked whether they felt confident in using Hypernote on completion of the assignment, the majority of students said that they were confident, although many had really only got to grips with the software during completion of the assignment.
Comments from tutors on the completed assignments were, not surprisingly, rather less optimistic than their students, and it appears that the scripts were variable in quality, with a few students producing good work, while others appeared to have put all their effort into mastering the software, at the expense of content and analysis.
"The middle of the road students..lost some of the content, value and relevance, while "tinkering" with the tool."
"The most important point for me is that it was not until [assignment] 5 that I understood certain parts of Hypernote. I think that this aspect of the course could be improved by providing some short activities to check that the student is on the right track."
This last comment is interesting because in fact the course had included a series of activities which were designed to build students' confidence in using the software. But the crucial fact was that students received no feedback on these activities, which were not integrated with the assessment strategy. Many students had either missed out the activities altogether, or otherwise assigned a low priority to them.
Such tasks as learning to use search tools, or conferencing software, or getting to know your fellow students online, take time and practice. There are probably two aspects to the role of formative assessment in these examples. The students appreciated the value of feedback, in order to check that they were "on the right track". There was also a motivational force, in driving student learning so that they undertook the activities set at an appropriate time, and in a conscientious way.
Of course, feedback which is designed to give students a running commentary on their progress must be delivered in a timely manner if it is to be effective and usable before the next assignment, and web based assessment may offer a solution to this progress and achievement testing (see for example Charman, 1999).
E-learners should participate in their assessment
If we are to develop self-direction in e-learners, and a more independent approach to learning, then it makes sense to allow them to participate more in the assessment process. It is after all a better reflection of the more open, student-centred courses which we are designing for them. This must be an important direction for assessing e-learners, and networking offers a number of opportunities, many of which are familiar to face to face teaching, with additional enhancements provided by the options to archive resources, and the advantage afforded by asynchronous discussion in providing time for reflection.
There are a variety of ways in which networking can be used to encourage student participation in assessment, and one direction is to provide enhanced formative feedback.
There are certain times when this is particularly appropriate. For example, we found that students on both courses described here valued comments on assignments, in particular at the start of the course. It appeared that they needed all the help they could get in these initial stages, in order to write appropriately, in an unfamiliar genre, for an unfamiliar course.
"At least now I have a feel for what to expect when it comes to writing these essays and so I hope that the experience of this first assignment may hold me in better stead for the future."
Using networking, feedback can be given to a whole tutorial group, forming the basis for online discussion and dialogue on interpretations of assignment wording or criteria, while at the same time offering some relief to the marker's workload. In this context, model answers have been used in order to give examples of effective writing or well structured assignments (see Barrett & Paradis, 1988; Macdonald, 2001). Building on this principle we have experimented with an "electronic scrapbook" of writing samples drawn from assignments written by students, as a way of using student resources to illustrate a number of teaching points and a range of writing styles. Such a scrapbook can be archived and used in subsequent presentations, although of course it tends to lose some of its "today relevance" if the scripts are not from recognisable students.
However, by definition such formative support is received retrospectively, and many students often feel that they would like more help before assignment submission, as these students describe:
"...it would have been helpful to provide a clear model of what was expected prior to the first essay, since you are clearly looking for a certain formula of response."
"I would have felt more comfortable with more explicit 'hints' as to whether my thoughts were moving in the right direction."
Often tutors will provide this kind of proactive support in traditional face-to-face tutorials held prior to an assignment, but for an online course it has to be given in other ways. At an informal level some students have used computer conferencing for peer support in assignment preparation.
"One aspect of conferencing which I find useful ...is the exchange of actual pieces of work. With modern technology it is simple to attach a file of work to a conference message for people to look at and comment on...It has proved very interesting and useful to be able to look at how other people have approached activities."
A natural progression is the concept of iterative assignment development, where assignments are submitted for formative review by the tutor, so that constructive comments can be used to develop the written work (Davis & Berrow, 1998; McConnell, 1999). This is after all the way in which academics operate when writing academic papers, and the process becomes very simple using networking. There are institutional constraints in operating such an exercise at scale, and it is not widely in use at the OU UK.
At a more advanced level, students might participate in assessment through peer review. Students can learn to comment on each other's work as an integral part of summative assessment, and this has been shown to be an effective approach when helping them to develop a critical approach to their written work (see for example Boud & Falchikov, 1989). The use of networking means that this is now feasible for distance learners (Macdonald, 2001), and gains the added advantage that scripts can be submitted anonymously, if that is appropriate. Peer review is a demanding task for undergraduates, because they need the confidence firstly to judge fellow students' work, and secondly to be able to give a critique without giving offence. However, with appropriate scaffolding to guide students, it is feasible, and has been demonstrated successfully this year with 200 students on a second level course at the OU UK.
Formative assessment of e-learning.
E-learning is enjoying wide popularity in Higher Education, and many courses are now adopting networked environments for delivery or tuition, making use of the potential for online collaborative learning and access to wider information resources in order to enrich course outcomes. These changes bring implications for the assessment strategy. However, I make no claim here that assessing e-learning is really radically different from assessing learning. The same principles apply. Basically, the key to supporting e-learning development will lie in an understanding of the complexity of the processes which we are asking our students to undertake.
Networked courses which adopt constructivist philosophy may impose new and unfamiliar demands on the students who study them. E-learning involves a complex mix of basic transferable skills and literacies, some of which are embedded within an understanding of a discipline. E-learning courses may expose students to a more demanding approach to study, requiring greater self direction and a critical approach to study than has been the norm in the early undergraduate years. All this means that it is important to support e-learning development, whatever the subject of the course.
The following comments summarise some suggestions on assessment design for e-learning, while recognising that considerations of validity and authentic assessment always need to be balanced against feasibility and marking reliability.
The formative assessment of e-learning courses has a variety of roles to play in supporting course outcomes, and ensuring that students are equipped to benefit from the rich learning environment. The assessment of e-learning need not necessarily be online, although there are times when that is appropriate, for example when rapid feedback is required on progress and achievement testing.
Constructivist approaches to study require students to learn by engaging in a variety of activities, which may involve collaborative work, problem solving or resource based learning. If such activities are central to course outcomes, then it is important that students undertake them conscientiously. Assessment strategy can be used as a way of affording students an opportunity for learning at critical points in the course. Since e-learning is such a complex process, students can benefit from incremental support through assignments which encourage them to build their skills gradually, and provide feedback on their efforts. If students are to be given greater autonomy in their networked study, then assignments which encourage greater student participation may help them to develop a self directed approach. Networks can be employed to deliver enhanced versions of innovative assignments used in face to face situations. For example, electronic scrapbooks, online peer review and iterative assignment development.
My grateful thanks to my colleague Dr Pete Cannell for his thoughtful contributions and critical comments on this script.
Notes on Contributors
JANET MACDONALD is E Learning Coordinator at the Open University in Scotland, where she is closely involved with the development of competence in students studying on networked courses. Email email@example.com
AMERICAN ASSOCIATION OF COLLEGE AND RESEARCH LIBRARIES (1999) Information literacy competency standards for Higher Education http://www.csusm.edu/acrl/il/index.html
Barrett, E & Paradis, J. (1988) Teaching writing in an online classroom Harvard Educational Review 58 (2) pp.154-171
Black, P. & Wiliam, D. (1998) Assessment and classroom learning Assessment in Education 5 pp.7-75
Boud, D. (1995) Enhancing learning through self assessment . London, Kogan Page
Bruce, C & Candy P (Eds) (2000) Information literacy around the world . Advances in programs and research . Wagga Wagga, Centre for Information Studies, Charles Sturt Univ.
Charman, D (1999) Issues and impacts of using computer based assessments for formative assessment. In: Computer assisted assessment in Higher Education . Ed S Brown, P Race & J Bull, SEDA/Kogan Page.
Collis, B. (1998) New didactics for university instruction: why and how? Computers and Education 31 pp.373-393
Collis, B, De Boer, W & Slotman, K (2001) Feedback for web based assignments. Journal of Computer Assisted Learning 17 pp. 306-313.
Davis, R. & Berrow, T. (1998). An evaluation of the use of computer supported peer review for developing higher level skills. Computers and Education 30 (1/2) pp. 111-115,
ECONOMIC AND SOCIAL RESEARCH COUNCIL (2002) Towards e-quality in networked e-learning in Higher Education, http://csalt.lancs.ac.uk/esrc/manifesto.pdf
Goodfellow, R. (2001) Credit where it is due: assessing students'
contributions to collaborative online learning. In David Murphy, Rob Walker
& Graham Webb (Eds) Online learning and teaching with technology , London: Kogan Page, pp.73-80.
Higgins, R Hartley, P & Skelton, A (2002) The conscientious consumer: reconsidering the role of assessment feedback in student learning. Studies in Higher Education 27 (1) pp. 52-64
Hill, J (1999) A conceptual framework for understanding information seeking in Open Ended Information Systems Educational Technology Research and Development 47 (1) pp.5-27
Knight, P (2002) Summative assessment in Higher Education: practices in disarray. Studies in Higher Education 27 (3) pp. 275-286
Lea M. & Street, B (1998) Student writing in Higher Education: an academic literacies approach. Studies in Higher Education 23 (2) pp.157-172
Lockwood, F. (1992) Activities in self instructional texts . Kogan Page, London
MCConnell, D (2001) Implementing computer supported co-operative learning (Kogan Page 2nd ed London.)
McConnell, D (1999) Examining a collaborative assessment process in networked lifelong learning Journal of Computer Assisted Learning 15 pp. 232-243
McDowell, L (2002) Electronic information resources in undergraduate education: an exploratory study of opportunities for student learning and independence. British Jounal of Educational Technology 33 (3) pp. 255-266.
Macdonald, J. (1999) Appropriate assessment for resource based learning in networked environments . Unpublished PhD thesis, Open University UK
Macdonald, J. (2001) Exploiting online interactivity to enhance assignment development and feedback on an ODL course. Open Learning , 16 (2) pp.179-189
Macdonald, J., Heap, N. & Mason, R. (2001) "Have I learnt it?" Evaluating skills for resource based study using electronic resources. British Journal of Educational Technology 32 (4) pp. 419-434
Macdonald, J. & Twining, P. (2002) Assessing activity based learning for a networked course. British Journal of Educational Technology 33 (5) pp. 605-620
Mason, R (2002) E-learning: what have we learnt? Improving Student Learning using Learning Technologies. Proceedings 9th Improving Student Learning Symposium, 2001 pp. 27-34.
Mason, R & Bacsich, P (1998) Embedding computer conferencing into University teaching Computers in Education 30 (3/4) pp. 249-258
Oliver, R. (2001) Exploring the development of critical thinking skills through a web supported problem based learning environment In Stephenson, J (Ed) Teaching and Learning online. Pedagogies for new technologies Kogan Page, London.
Perry, W G (1970) Forms of intellectual and ethical development in the college years. Holt Rhinehart and Winston, New York
RONTELTAP, F. & EURELINGS, A. (2002) Activity and interaction of students in an electornic learning environment for problem based learning. Distance Education 23 (1) pp.11-22.
Salmon, G. (2000) E moderating. The key to teaching and learning online . London, Kogan Page.
Schrage, M. (1990). Shared minds: the new technologies of collaboration . New York: Random House.
Thomas, P & Carswell, L (2000) Learning through collaboration in a distributed education environment. Educational Technology and Society 3 (3) pp.1-15