Education-line Home Page

The assignment that triggered change:
Assessment and the relational learning model for generic capabilities.

Sylvia Lauretta Edwards and Christine Susan Bruce
Centre for Information Technology Innovation

Paper presented at the Learning Communities and Assessment Cultures Conference organised by the EARLI Special Interest Group on Assessment and Evaluation, University of Northumbria, 28-30 August 2002

Sylvia Lauretta Edwards is a 2002 QUT Teaching Fellow, and a Lecturer in the School of Information Systems at Queensland University of Technology (Australia), lecturing in WebSite design, e-commerce, and information resources. Sylvia's research interests include WebSite design, the Internet, lifelong learning and information literacy. She is currently working towards her Doctorate in IT aiming to determine variation in the ways students approach information searching and to make sense of the students understanding of information searching and retrieval. She is regularly invited to speak at conferences on Internet Searching and WebSite design and provides considerable consultancy work for state and national organisations requiring Internet training.

Christine Susan Bruce is Associate Professor and Director of Teaching in the Faculty of Information Technology at Queensland University of Technology (Australia). Her teaching areas include information organisation, information user instruction, professional practice, information management and research methods. She has extensive experience in information literacy education and research, as well as broader interests in the perceptual worlds of information and technology users, and of information technology researchers. She has written over 60 articles and conference papers in this area and consults to universities across Australia on the curriculum integration of information literacy. Her most recent book, co-edited with Professor Philip Candy, is titled Information Literacy Across the World: Advances in Programs and Research.

Address for communication

Ms. Sylvia Edwards Business Ph: +61 7 3864 2759
Lecturer, School of Information Systems Business Fax: +61 7 3864 1969
Faculty of Information Technology, Email: s.edwards@qut.edu.au
Queensland University of Technology,
Box 2434, Brisbane 4001, AUSTRALIA

ABSTRACT

In a context where changes in conceptions or experiences associated with learning generic skills is a desirable learning outcome, how can assessment instruments be designed to bring about the desired changes? In this paper we show how understanding variation in students' experience of learning a specific generic capability represents the first step in designing assessment instruments for bringing about desirable learning outcomes. Our research has revealed that from a student's perspective two major elements produce changes in conception or experience. From a teacher's perspective, these changes should lead the student into desirable learning outcomes. The two elements identified by students are assignments designed to encourage reflection and the ability of the teaching staff. On the basis of student perceptions, therefore, we are further challenged to carefully construct assessment to bring about change.

Introduction

In a context where changes in conceptions or experiences associated with learning generic skills is a desirable learning outcome, can assessment instruments be designed to bring about the desired changes? Over the past few years, observing and researching students' learning to search the Internet has been enlightening. In research interviews with the students, individuals have commented on the importance of planning a search and having the ability to use the different search tools. They mention also the importance of being able to search using the searching options available to filter results into a small but highly relevant set. Words like Boolean operators, or truncation, have rolled off students tongues during interviews, and the teacher has been impressed by the students' ability to "talk the talk". Unfortunately, later, while seated at computer terminals, some of these same students have not "walked the walk". At the terminals during the research interviews, or observing them later in classes, some of the students use none of the tools they mentioned, they ignore or fail to notice typing errors, and then express frustration about the tool's poor design, promptly switching to another search engine and simultaneously switching search terms. This polarity between students' words and actions is concerning, suggesting that their experience needs closer investigation. This led to an ongoing research project investigating students varying experiences of searching for web-based information.

The students who participated in the research have all been enrolled in a QUT (Queensland University of Technology) FIT (Faulty of Information Technology) subject ITB322 Information Resources. In this unit students learn to identify, retrieve and evaluate print and electronic business information resources that are relevant to a variety of problems; thereby applying their knowledge in Internet, Intranet, and virtual library environments. Primarily the unit is designed for students to learn about a variety of information resources and their uses, independent of the format of those resources. The curriculum focuses on information retrieval techniques, user needs analysis, databases, commercial information providers, traditional published sources (monographs, journal articles, patents, standards, and government documents), people, and the Internet. Students are therefore exposed to information resources which will be useful in their future careers.

This paper shows how understanding variation in students' experience of learning a specific generic capability can represent the first step in designing assessment instruments for bringing about desirable learning outcomes. There is ample research suggesting that learning activities can be designed to improve learning outcomes . Marton further proposes that the learning study is a powerful tool in developing learning activities that bring about desirable learning outcomes, and Biggs reminds us of the principle of constructive alignment. We agree with Biggs and Marton that learning activities, including assignments, can be crafted to bring about desirable learning outcomes.

In our application of the relational approach to teaching and learning to the generic capability agenda, an attempt has been made to design formative assessment instruments to construct the student's experience of learning. It was found that students, during interviews, suggested that it was assignments that triggered a change in their experience of information searching. We determined, therefore, to pursue the use of assessment as a mechanism for opening up the space of variation associated with learning to search the web. This means that we are suggesting that assessment can be constructed to enable the students to move from a position of seeing the world in one, or a limited number of ways, to seeing the various lenses available through which to view the world.

The work began with an investigation of students' different ways of searching the web. The findings from this work led to an analysis of assignments and recommendations for further redesign of those assignments. Based on this evidence, this paper discusses how assessment may be used strategically to construct students' experience of learning. We suggest that the process of understanding how students learn to search the internet, and crafting assessment so as to construct appropriate experiences, may be applicable to teaching and learning other generic skills.

In order to understand the background for our research, existing research into web-searching behaviour should be briefly outlined.

How do students experience Internet searching

Most research into web-based searching behaviour currently falls into two broad categories. The first of these could be described as the classic information retrieval research. The second is more complex, but basically could be described as research into internet searching or internet use.

Growing out of information retrieval or information management roots, the classic research area of information retrieval traditionally has considered database, OPAC, and other information retrieval system searching. It has been studied in depth by numerous researchers to date. The primary focus is often related to the design of the retrieval system or the likely relevance of the final results . Studies have commonly considered what has become known as the recall or precision ratios, which look at the number of relevant and non-relevant documents retrieved compared to the number of relevant documents actually available . In general, while it is possible from this research to identify trends in typical interactions between searchers and the system , these studies lacks an understanding of why the searcher actually performed the action. In the end they quantify search behaviour, and work to describe or seek the meaning embedded in the behaviour.

Researchers have also been looking at many aspects of web-based searching behaviour, from pure database searching and likely information retrieval , to search strategies and information seeking in context approaches . Typically some form of quantifying measure has again been used in most of these approaches and in general, attempts have been made to describe the "average web-based searching behaviour". We believe the challenge is to identify the non-average searcher.

Human factors in web-based searching behaviour should not be ignored. We need to place the end user firmly in the centre of the picture. It is not solely about recall and precision ratios, it is not solely about their likely computer literacy skills, it is not solely whether or not the interface or the system tool is well designed, and it is not solely about the cognitive abilities of the end user. It is a combination of factors, and we should be attempting to look for the variation in the experience of every type of internet searcher. In doing so, we may identify why particular search behaviour is evident, and in understanding the underlying reasons for the approach, we may be able to see methods to build bridges to move from one approach to another, and to build a framework for people to move into a more satisfying search experience.

Based on these findings, our research project has a series of broad aims. Those relevant to this paper include:

1. To determine variation in the ways IT students approach information searching when using the Internet and library databases.

2. To determine variation in IT students ways of learning to search for information when using the Internet and library databases.

3. To recommend teaching and learning strategies for curriculum design that are based on managing student's experiences.

The findings reported in this paper relate to the third aim, and specifically reveals what the students themselves perceived in their information searching experience based on the work undertaken in the subject unit. The research study findings for the first two aims are briefly outlined below.

Describing different ways of internet searching

Based on the analysis of the experiences of upper undergraduate and post-graduate IT students four categories have been identified that capture students' different ways of searching and learning to search for information. These categories can be shown in the outcome space provided in Figure 1. The outcome space reveals a logical path for Categories 2 through to 4 and a major gap between Categories 1 and 2. The research findings suggest that assessment design as well as reflective opportunities for students is required to bridge this gap. In the following section, the four categories are briefly outlined. Further information on the categories can be found in Edwards and Bruce .

Figure 1: The Experience of Information Searching: The Outcome Space Sept. 2002

Each of these categories is associated with different meanings being assigned to the search experience. Each is also associated with different awareness structures, different approaches to learning and different search outcomes. The brief descriptions provided here were developed to answer the following questions: What was the focus for students in the experience? Were they aware of the information environment and its structure? Was there any evidence of planning or any ongoing reflection used in their approach? These questions emerged as keys to describing this phenomenon during the course of the analysis.

Category 1: Information searching is seen as looking for a needle in a haystack.

In this category students see information searching as similar to looking for a needle in a haystack. A significant amount of attention is directed towards the topic. They appear to see it as imperative to understand the topic or they will "never find it out there." Although they are aware of the information environment they have no appreciation of the importance of neither the structure of that environment, nor the structure of the tools that they use to find information. More importantly in this category there is little evidence of approaching the search process in a reasoned or a reflective manner. There is usually an assumption that the information required is not available at this source, or the tool in use is of poor quality and does not index the required information.

Category 2: Information searching is seen as finding a way through a maze.

In this category students see information searching as the process, or the planning, of a search. While they still focus on the topic, there is a strong emphasis appearing on the choice of terms and synonyms, databases, and retrieving results into a useable format for later work. The process or the planning of the search has become more important, with students beginning to use advanced search features, and talk about some aspects of the quality of the information found. In this category they are more likely to persist, consider alternatives, and persevere to find results. However, again there is still a tendency to blame the tool rather than question their own abilities.

Category 3: Information searching is seen as using the tools as a filter.

In this category students see information searching as using the tools as a filter to find information. They tend to use the tools to help them understand the topic as well as to find the required information. They are much more aware of the structure of each of these tools and show an ability to adapt their searching based on the tool they are currently using. In this category students take the necessary steps to correct mistakes as required and planning is evident. This planning often includes an analysis of the terms and a more pronounced attempt to identify synonyms before proceeding. There are also attempts throughout the search process to identify and change strategies based on the results of the first attempts.

Category 4: Information searching is seen as panning for gold.

This category could also be described as using the tools as a filter, but this time the intention is to limit results to high quality information. In this category students see information searching as a process of using the tools during the search to limit the final set of results to include only the highest quality resources. The intention is to use the appropriate tools to find only primary information resources. As the awareness of primary and secondary information is heightened, the tools and their structure are used to both refine the topic and refine the search, to help filter out poor quality items. Strong planning and reflection are evident and the searching process includes changing strategies based on the results of first attempts.

Students' views of influences on their experience of learning to search the Net

It can be seen from the four categories defined that students have different experiences when searching for information. Secondly, and more importantly, at the second research interview, conducted after the study of the subject was completed, students reflected upon their evolving searching experience. Each student perceived that they had changed their experience of searching compared to their experiences prior to the unit commencing. Each student believed that their searching experience had improved in both speed and the end quality of the results found. More importantly, most students stated an ability to transfer the knowledge gained to other units of study and to help their peers when they searched for information. The students identified two reasons for change. The two reasons identified were assignments designed to encourage reflection and the ability of the teaching staff. In the following section, the students reasons for the assignments identified will be briefly given, and their comments on the quality of the teaching staff outlined.

Assignments Designed to Encourage Reflection

There were three items of assessment designed for the unit. The three items and their relative assessment weight for the unit were:

1. Reflective Journal (20%)

Each week students are asked to complete a Journal entry as a means of encouraging interaction with the unit's material. The students are asked to critically reflect upon and summarize the content of lectures, tutorials, set readings, and field visits. They are asked to consider their information searching experiences using a reflective approach .

2. Information Consultants' Search (30%)

For this assignment the students are given a choice of four topics, and they are asked to assume they are working for a client. The final product is expected to be 10 to 25 highly relevant items from a series of available information resources, and a search report of this process.

3. Information Resource Guide (50%)

This assignment, a team project, asks students to produce a Resource Guide to a subject area that they may like to work in on completion of their Degree. The team is to list only the best of the major resources that would be required to solve most questions encountered when working in area. Group presentations of the "Top 5 Resources" found are required.

Further information on the design process may be found in Edwards .

Students attributing changes in their information searching experience to assessment items identified the reflective element of two of the assignments as important.

The first assignment identified by students was a weekly reflective journal. Fifteen percent (15%) of the students attributed their change in searching method to the reflective journaling which was undertaken weekly. Students stated that the effort of having to reflect upon the content each week had caused them to change their experience.

... if I wasn't doing the journals this semester I don't think I would have rethought a lot of the things that I'd learned straight away, and I think that a lot of the good things that I'd learned may not have, you know, may have just disappeared.

Male student, UG 21-25 yrs (Int 2001:2.2 p.8)

Please Note: (Int 2001 1:2.2 p.8) = 2001 Interview: Interview Number 2: Participant 2 (p.8 of transcript).

The second assignment highlighted by the students was the Information Consultants Search. A further thirty-five percent (35%) of students attributed their change to this assignment where they were required to produce a search report of their search process. The students reported that being forced to reflect and report on assumptions, keyword selection, or other aspects of the search process, made them realise that they should change the way they approached searching.

What if I just blame it on assignment two ... That - big time change for me. Well you see it had to be comprised of all these different components. Like ok, your client ...synopsis, your assumptions, your keywords and synonyms, your ... there were so many different areas. .... So you had to make sure you that you know all these in order to put down in your report if you really wanted to do well in this report. ... So I came up with all these assumptions and was like, oh ok I've never thought about these before.

Female student, UG 21-25 yrs (Int 2001:2.3 p.11)

Quality Teaching Staff

The remaining fifty percent (50%) of students attributed the changes in their searching experience to great teachers who made the unit fun. They stated that if the lectures were not fun and interesting, and the staff not caring and approachable, then they would not have learnt anything. In their opinions, the great teachers each had a fun approach to classes, as well as being eager to help and simply were approachable people. In their view this was the reason they changed. They each said that either the lecturer, or the tutor, was the best teacher that they had ever had.

... Like they were trying to be inventive and creative and trying to make it interesting. And it really worked. All the students noticed that. And so, like you know, we'd talk about the lectures... and that made learning more enjoyable and that made us pay more attention when they were talking, it made us attend tutorials. Because they say stuff that is interesting. They don't just read the notes.
Male student, UG Under 20 yrs (Int 2001:2.4 p.8)

Using assessment to craft students experiences of learning to search the Net

It is important to note that the primary focus in the design of the assessment items was to develop three generic attributes fundamental to successful learning in the IT discipline; the development of information literacy skills, communication skills, and the ability to work in teams. Although the design intent was to encourage the development of generic attributes, it would now appear that an intuitive element was also included; the design for information literacy skills has included skill development across some of the categories of information searching.

The students' views about the role of assessment reinforced for us the potential role of assessment in crafting students learning experiences. We subsequently analysed the early assignments in relation to the categories of description found. We wanted to identify the ways of experiencing Internet searching that each item of assessment encouraged? Further, we wanted to determine how the assessment could be further developed to craft student experiences?

Analysing the existing assessment in relation to different ways of seeing

Based on the research findings, we have mapped the assessment items against the categories of description to identify how well the assignments have been designed to encourage learning. The results of the analysis and the students' perceptions are shown in Table 1.

The principles upon which we chose to analyse, and later to redesign, the assessment for the subject include the assumption that learning a generic capability means coming to experience that capability differently. Assessing a generic capability, therefore, means discerning the character of the learning outcome achieved by the student. Assessing a generic capability also provides the opportunity of designing the assessment instrument in order to influence the character of the desired learning outcome.

From the initial analysis, in Table 1, it appears that Category 3 is built into the assignments very clearly. It can be seen that both Assignment 1 and 2 mapped with Category 3, in that both assignments require the students to reflect upon the searching process. This built in reflection requires the students to experience searching using a reflective approach. Assignment 2, in particular, asks students to refine their topics, devise alternative search terms, and adapt their searching based on the tool in use. These are the experiences of Category 3 searching.

In Table 2 each assignment is analysed to consider how they encourage the experience of each of the categories. It can be seen from Table 1 and 2 that in both Assignment 2 and 3, to a limited extent, there was a requirement for the students to limit to high quality resources, or to have the experience of Category 4 searching. Unfortunately, there was only a limited requirement in either assignment for students to reflect on this process.

Recommending changes to the assessment

Based on the analysis, it can be seen that Assignment 2, which required a limit to high quality resources only, and Assignment 3 which asked for the "Top 5 Resources", need to be further enhanced in order to encourage the students into a deeper experience of learning to search for information. The simplest step here would be to increase the relative weighting of criteria marks to encourage the student to both understand the necessity of searching for quality resources, but to also encourage the student to reflect upon this process in a search report.

Put simply, for each category we have parts. These parts can now be used to craft the assignments further. In Table 2 simple examples are given for both Assignment 2 and 3 to include a series of questions designed to encourage reflection equivalent to the experiences of Category 4 searching. In this way, as with the existing Assignment 2 Search Report, the reflection element can be built into the Report Criteria.

It would appear that while there is a need to design the assessment to make the assignments work harder, to lead students into the structure of the experience that is considered desirable, the existing assessment does encourage the students to move into a higher level of information searching experience. These elements should be maintained, but the assessment needs to be crafted to work harder. On the basis of student perceptions, we now need to carefully re-construct assessment to bring about desired learning outcomes to encourage reflection to Category 4.

What we have tried to achieve in modifying these assignments is the simultaneous attention of students to relevant dimensions of the searching experience. That is, we are trying to allow the student to be aware of what is happening in many areas of the experience. The structure of awareness section of each category reveals the elements that need to be attended to in designing the assessment. In other words, assessment can be designed to encourage students to attend to the different aspects they need to be aware off. In the example reported here, reflection to high quality information resources needs to be further enhanced to allow the students to experience that aspect of Category 4.

Conclusion

Our research has revealed that from a student's perspective two major elements produce changes in conception or experience. From a teacher's perspective, these changes should lead the student into desirable learning outcomes. The two elements identified by students are assignments designed to encourage reflection and the ability of the teaching staff. On the basis of student perceptions, therefore, we are further challenged to carefully construct assessment to bring about change.

Finally, while embedding generic capabilities into assessment is vital to trigger changes in conceptions or experiences, and to deepen the learning experience, the ability and quality of the teaching staff should not be ignored. Their ability, to arouse curiosity and make the learning environment both stimulating and enjoyable, helps promote interest in the topic, which may also lead to changes in learning outcomes.

References

BIGGS, J. (1996). Enhancing Teaching through Constructive Alignment. Higher Education , 32(3), 347-364.

BOWDEN, J., & MARTON, F. (1998). The University of Learning: Beyond Quality and Competence. London: Kogan Page.

EDWARDS, S. L. (2000). You Have Provided Me With A New Set Of Tools And Taught Me How To Use Them: Embedding Generic Skills within the IT Curriculum. Paper presented at the Lifelong Learning Conference: Inaugural International Lifelong Learning Conference, 17-19 July 2000, Yeppoon, Central Queensland, Australia.

EDWARDS, S. L., & BRUCE, C. (2002a). Needles, Haystacks, Filters and Me: The IT Confidence Dilemma. Paper presented at the Lifelong learning: building learning communities through education. 2nd International Conference , 16-19 June 2002, Rydges Capricorn International Resort, Yeppoon.

EDWARDS, S. L., & BRUCE, C. (2002b). Reflective Internet searching: an action research model. The Learning Organization: an international journal , 9(3/4), 180-188.

FIDEL, R., et al. (1999). A Visit to the Information Mall: Web Searching Behaviour of High School Students. Journal of the American Society for Information Science , 50(1), 24-37.

JANSEN, B. J., & POOCH, U. (2000). Web user studies: A review and framework for future work. Journal of the American Society of Information Science and Technology , 52(3), 235-246.

KLOBAS, J. E., & CLYDE, L. A. (2001). Social influence and Internet use. Library Management , 22(1/2), 61-67.

MARTON, F., & BOOTH, S. (1997). Learning and awareness . Mahwah, N.J.: L. Erlbaum Associates.

MARTON, F., HOUNSELL, D., & ENTWISTLE, N. (Eds.). (1997). The experience of learning: implications for teaching and studying in higher education (2 ed.). Edinburgh: Scottish Academic Press.

MARTON, F., & TSUI, A. B. M. (Eds.). (In press). Classroom discourse and the space of learning. Mahwah: Lawrence Erlbaum Assoc.

ROBERTSON, S. E., & HANCOCK-BEAULIEU, M. M. (1992). On evaluation of IR systems. Information Processing and Management , 28(4), 457-466.

SILVERSTEIN, C., HENZINGER, M., MARAIS, H., & MORICZ, M. (1999). Analysis of a very large Web search engine query log. SIGIR Forum , 33(1), 6-12.

SPARCK-JONES, K., & WILLETT, P. (Eds.). (1997). Readings in information retrieval. San Francisco: Morgan Kaufman.

WILEY, D. L. (1998). Beyond Information Retrieval: Ways to provide content in context. Database , August/September.

This document was added to the Education-line database on 21 October 2002