Title: The Impact of Information Skills Training on student learning: Do we make a difference?

Research Team: Wendy Bastable, Liz Bellingham, Steve Homer, Gill Hughes, Vince Mcnutt, Oliver Pritchard

Department of Learning Resources

Background and rationale

This project aimed to test the impact of Information Skills sessions provided by librarians within the Department of Learning Resources at the University of Wolverhampton. This work is usually seen as a "good thing" and informal feedback has suggested that the sessions are valued by both students and academic staff. A review of the literature suggested that there is little evidence of the evaluation of this work within academic libraries, with one survey suggesting that only 8% of respondents had undertaken such a review.

We wanted to gain an impression of the influence and worth of the sessions with students and academic staff. In particular we aimed to establish whether students did acquire skills and knowledge from the sessions and whether these were applied effectively in their approach to academic study and assignment work.

The Research

The project involved three randomly selected groups of students in three different schools and subject areas: Humanities, Languages and Social Sciences (Sociology), School of Health Studies (Complementary Therapies), School of Education (Education). The study involved a total of 41 students in the initial session with smaller, self-selecting numbers volunteering for focus group activities.

The librarians who normally work with these subjects provided standard sessions to each group. It was important that the sessions were allowed to include normal variations between subjects to allow the identification of best practice and weakness.

Four strands of research were then applied to each group. These were: pre-session questionnaire, post-session questionnaire, student focus group and interviews with module tutors. This design aimed to test the progression of learning and any application of skills and knowledge over a period of time after the initial workshop session.

At each stage the results were analysed to give a picture of the student perception of their own learning and the views of module tutors as to the longer term impact of the sessions. In conducting the research in this way we were seeking to identify impressions and trends in the learning process based on a small but randomly selected group of students.

The Outcomes

Pre – session questionnaire – all groups

Overall these results (Appendix 1) suggest a level of uncertainty and lack of confidence among participants. It is important to recognize that the groups involved level 2/3 students who were still expressing uncertainty some way into their course of study; in fact, more than 50% of students expressed some degree of doubt about knowledge and/or ability.

Exceptions to this are found in questions 7 regarding saving information from Web sites and question 8 on using the internet to find subject information. However, a total of 45% also responded negatively to question 7 which might be considered a standard ICT skill.

Less than 50% of participants agreed that they had the skills to use electronic resources. Only 25% were confident in evaluating web sites and only 27% were familiar with the Learning Resources Web site. There was also uncertainty about the location of electronic full text journals with over 70% expressing some level of doubt. Similar levels of response are evident for question 6 about the use of online indexes.

On the use of the On-line Public Access Catalogue (OPAC) two questions were asked. One tested the use of a Reading List option that allows students to access module reading list resources held in the Learning Centre. Over 50% were unsure or disagreed. Question 2b asked a more general question which sought to test confidence in using the OPAC as a subject search tool. Again responses were negative in this area with some 68% expressing doubt or disagreement in their response. This is interesting given the level of study and the importance of OPAC as a key finding tool.

The highest levels of confidence expressed were in response to question 8. Participants felt comfortable with their ability to search the Internet for relevant information. However not all would be confident in evaluating the sites and information retrieved. This is an important issue given the variation in quality and quality control found on Internet sources.

Post – Session Questionnaire – All groups

Following the workshop participants were asked to complete a second questionnaire by way of evaluation. This was designed to test the general suitability of the session and to cover some of the general themes exposed in the pre-session questionnaire. The post session exercise was by design, not intended to mirror questions asked before the session. The rationale underpinning this decision was that this research was to determine the longer-term impact of our work. Responses immediately after the session might have been positive but would not necessarily have had great value. In addition the research team wanted to explore new areas about the delivery and content of the sessions.

The overall picture of the session content, design and delivery is a positive one (Appendix 2). Perhaps most significant is the apparent rise in confidence expressed about the use of sources. While the design prohibits any detailed analysis here, question 3 reflects a rise in confidence in the use of on-line sources covered in the sessions. This is further supported by the positive response to question 5 (All students need the session). Participants felt strongly that on-line searching was important and that a structured introduction was of value. However, 24% of participants were still unsure at the end of the session. This contrasts with the strong response to question 4 (clarity of session delivery). There could be a number of underlying explanations (e.g. the need for further practice and experience) for this and we note that that this result was strengthened by the Education group where the need for more practical work was identified as an issue by participants.

Overall a number of participants had found some useful subject databases during the session. This is an important finding given the low levels of confidence expressed about the use of subject databases in the pre-session questionnaires. This is an area where Librarians can offer a subject information specialism which may be unique within their institution. In addition there were positive responses made about the practical hands-on elements and the use of search engines. The latter is possibly significant. In pre-session questionnaires internet subject searching was consistently the highest level of response. However a number of those who responded to this question (19%) felt that a review of search engines had been a useful element.

Focus Group Results

In each group students were asked to identify elements of the sessions which were, for example most and least useful, key points learnt and what they had subsequently applied. This took the form of a structured brainstorm. Once ideas had been exhausted the group ranked them on an individual basis within the agreed group of identified ideas. In reading the results we must be aware that participants were being forced to make choices from a set of ideas which were already thought to be important. A low score does not mean that this element is not important but rather that it was judged less important in relation to other areas.

Education students (Appendix 3 Table A) valued finding full text articles on-line, closer definition of internet searches and use of OPAC. These were also identified as the key points learnt during the session and as elements applied after the session.

Sociology students (Appendix 3 Table B) were highly focused on web search skills and sources and cited these among the most useful elements. They also highly rated the use of an Internet Web Evaluation tool "Detective". Key learning included introduction to specific web pages, again suggesting that students want/need focused strategies for using the web. All students had applied at least one element of the session including specific search engines and the use of subject web pages .

Interviews with module tutors

The third (and final) stage of the research involved meetings with the module leaders of the students who had attended the original informal skills workshop. The aim was to test the longer-term effectiveness of the sessions and their impact on learning and also to assess tutors’ perceptions of the workshops and their value. This was an extremely important element in the project, as it looked to test the thesis that our work had some impact on the way learners approached and completed assignments.

The interviews were based around a questionnaire which was completed by the course tutor during a discussion with each of the librarians who ran the original workshop.

Complementary Therapies

(Appendix 4 Table A)

The tutor reported a change in students’ assignments after the session and ranked some of this trend to be a marked change. In particular the tutor noted that there had been a noticeable change in the improvement in the quality of resources used and that there had been some increase in the range of resources used.

There was slightly more use of journals, web sites, improved citation of sources and a more marked awareness of research-based material. The tutor commented that students were "more able to access and recognise quality sites". The grades for the assignment were slightly higher than previous work. Although it is clearly not possible to conclude that this improvement is wholly due to the use of information resources, the improvement in this area is visible and may have therefore contributed to the improved grades. In one case the grade had improved significantly. This relates to a student with special communication needs and the tutor commented that the work of this student had shown a "significant improvement" since the session.

The experience of this small cohort of students suggests a link between the information skills session, their approach to learning and their success in completing assignments.

Education

(Appendix 4 Table B)

The tutor reported a change in grades between assignments and that generally with a few exceptions, these were slightly higher. However this was based on the comparison of student performance in the current year with the previous one. By definition this would have involved two different groups of students. As a result it is impossible to draw any conclusions about the impact of the session on assignment grade.

Within this cohort however there are indications of value. The tutor commented "Even if we try to encourage them to use a range of printed and online resources (besides books) without a practical (specific) session, they don’t use them. When I was reading the assignments I could identify those who had attended the session."

This is a significant comment, which again suggests a value in the sessions and the skills they aim to deliver.

The tutor also commented that the skills gained could be evidenced in other group presentations where on-line sources were used to support argument. In particular in the education group we see the permeation of on-line sources to compare criminal statistics and underpin an oral presentation. This perhaps suggests that the skills gained are transferred into other areas of work and do underpin learning in the longer-term.

Benefits

This study brings a number of benefits. We have been able to assess our approaches to the design and delivery of information skills sessions. We now know for example, that working with OPAC, effective search techniques, web evaluation and practical exercises should form part of any session. This impacts directly on the quality of the student learning experience.

The study also suggests that the sessions are a valuable part of the learning process which offer new skills and experience. In particular, it suggests that the impact of the training has some longer term value and can influence approaches to academic study and achievement. This is valuable information in the context of both student retention and the current development of the Key Skills agenda within the University.

Evaluation

This research is in itself an evaluative exercise based on both student and tutor perceptions and comments. The results have suggested some improvements in grades before and after the workshop sessions. We have also been able to evaluate the strengths and weaknesses of this work and will improve future practice as a result.

Future Developments

This small study offers a future opportunity for a wider project involving a larger sample of students. The findings will be embedded in to our future practice in this area. We will seek to ensure that all sessions cover OPAC, focus on web evaluation, on good practice in building effective search strategies and the importance of highlighting subject databases and gateways. Finally we would want to ensure the proper evaluation of all future sessions to provide a basis for on-going improvement.

Appendix 1 : Pre-Questionnaire Overview – all subjects

41=100%

Appendix 3: Table B: Sociology Focus Group Results

Appendix 4: Table A: Complementary Therapies Interview with Tutor

Appendix 4: Table B: Education Interview with Tutor