Information literacy in the context of Evidence-Based Medicine: teaching and learning assessment of a course intended for 4th year students in medicine at the University of Liège (Belgium)

Introduction

At the University of Liège, an Evidence-Based Medicine (EBM) course intended for 4^th year students in medicine is focused on information search, selection and evaluation. This course is held in the context of a workshop in pharmacology and is divided into three activities of 2 hours each. Given the number of students (about 140 per year), the 3 activities are run four times for groups of 35 students. The first meeting provides a theoretical introduction to EBM and Medline searching. Then the students have a couple of weeks for small group work on a clinical question. They have to apply the principles of EBM in order to find, summarize and comment on a controlled trial comparing the efficacy of different drug classes for the treatment of respiratory diseases (asthma or chronic obstructive pulmonary disease). During the second meeting, each small group returns for a twenty minute oral presentation of arguments for or against the drug class they have studied. The last meeting consists of an advanced course in Medline searching through the online OvidSP portal.

Questions/Objectives

Two questions were asked in the present study: 1) Do EBM classes improve students' information literacy? 2) Are the teaching methods appropriate?

Methods

In 2011-2012, the first question was addressed by running the validated Fresno test (1) both before and three months after the EBM course. The Fresno test is an instrument that evaluates four EBM steps and that has strong psychometric properties (2). The test begins with the presentation of two clinical scenarios and includes 12 questions (open-ended questions, fill-in-the-blank questions and calculations).  Students are requested: 1) to formulate a clinical question for each scenario; 2) to name types or categories of information sources useful to answer the questions, and to describe their strengths and weaknesses; 3) to identify the best research design for one of the two scenarios; 4) to explain the strategy they would use for Medline searching; 5) to cite the criteria to be considered for critical appraisal of an article (relevance, validity and magnitude of effect size). The last five questions are related to mathematical calculations and best evidence for diagnostic and prognostic issues. These questions were pre-tested, but were removed from the post-test because the corresponding topics had not been taught and the students were not expected to have progressed in these areas.      

The relevance of teaching methods was evaluated through a survey administered after the post-test. Students were invited to express their opinions about the course.

Results

109 out of 154 enrolled students responded to both the pre-test and post-test. A sample of 25 students was considered in this paper corresponding to the first 25 names on the alphabetical registration list. A total of 50 pre- and post-tests was scored independently by two reviewers in order to assess inter-rater reliability. The maximum possible score for the first 7 questions of the Fresno test is 168 (24 points for each question).

18 students performed better in the post-test, in comparison with the pre-test, with scores increasing by between 1 and 36 points. 3 students obtained the same final score, and 4 others obtained lower scores in the post-test with a decrease of 3 to 20 points.    
In the present study, for the pre-test, the lowest score was 9 and the highest was 66. For the post-test, the lowest value was 17 and the highest value was 83.

Here, the average score was 41 for the pre-test and 50 for the post-test.

The sum of scores obtained by the 25 students for each question showed some improvement on the following topics: elaboration of a focused clinical question (240 to 266; +26 points), selection of the most appropriate clinical study design (81 to 219; +138 points), strategy for Medline searching (251 to 312; +61 points) and validity assessment (113 to 202; +89 points).

Responses to three questions obtained lower scores in the post-test: strengths and weaknesses of different sources of information (258 to 186; -72 points), relevance (34 to 30; -4 points) and magnitude of effect size (49 to 24; -25 points).

23 students out of the sample of 25 answered the satisfaction survey. All the respondents thought that the training on Medline searching should be scheduled before the oral presentation in order to support team working. The majority considered that the workshop objectives (14/23) and instructions (18/23) were clear as well as the user guide provided to help in the preparation of the oral presentation (19/23). 17 students acknowledged that they had not been motivated to follow these sessions. 21 recognized that the teachers were ready to help but only 8 thought that the teachers made the sessions attractive. 13 students considered that the EBM course would be useful for their studies and 20 that it would be useful for professional life.

Discussion

In answer to the question “Do EBM classes improve students' information literacy?”, we found that two-thirds of the students managed to improve their scoring in the Fresno test, even though their overall performance remained barely satisfactory. Several elements could explain these results: 1) the scores might have been better if the post-test had taken place directly after the last training session instead of three months later, and if the participation of the test were not just optional but taken into account for certification; 2) student progression might have been more striking if the students had been inexperienced at the time of the pre-test; however, they had already completed an introductory course in information retrieval two years earlier; 3) the severity of scoring.

Surprisingly, students were less successful in the post-test on the question related to strengths and weaknesses of different information sources. Perhaps they focused attention on issues related to EBM, and disregarded the question on library resources?

Regarding the severity of scoring, the following elements should be highlighted. For question n°1 (translation of clinical scenarios into answerable questions), no point was attributed if students provided keywords on a PICO grid instead of writing a full sentence.
Writing a complete sentence is certainly more complex than just citing elements. In some cases, students provided both answers: the different ingredients of the PICO grid were good, but the sentence did not match the PICO grid.           
In order for students to answer questions n°5 (relevance), n°6 (validity), and n°7 (magnitude of effect size), we decided that it was mandatory to clearly distinguish between the three criteria. However, in their assessment of the reliability and validity of the Fresno test in EBM, Ramos et al. (1) state that overall justifications made by respondents may be acceptable where they describe, for example, issues of relevance in answers to any of these three questions.       
It was very helpful to have two reviewers sharing opinions in the marking process, since the Fresno test is not easy to score despite having a scoring template provided by the authors. Numerous elements must be considered and raters must also refrain from interpreting students' ambiguous responses. As noticed by Shaneyfelt (2) and by Lewis et al. (3), time and expertise are required to score the test. For all these reasons, we consider that comparing scores assigned by different examiners in different settings should be undertaken with caution.

The survey highlighted an important organizational change that has to be made in the future (moving the Medline training session to before the oral presentation); it also reported students' overall satisfaction with the support offered for team working. Nevertheless, the number of students per group is much too high, meaning that they have to divide the work into parts and share responsibilities. Very few participants involve themselves in the different activities offered by the EBM classes: writing a clinical question, searching on Medline, selecting relevant papers, reading and reviewing a clinical article, giving an opinion and presenting results to others. As teachers, we shall have to adjust our evaluation grid in order to take into account individual participation in the different tasks.

Ideally, much more time should be devoted to the EBM course, in particular to extending the information search in databases other than Medline. But the curriculum of the degree course in medicine is already overwhelming. An alternative would be a transversal integration of EBM activities into other courses in order to improve students' knowledge and skills without increasing the workload.      

As a conclusion, we should remember that 20 out of the 25 students surveyed think that the EBM course will be useful for professional life. This is rewarding for librarians, who consider it a part of their mission to ensure medical students gain the proper tools for EBM practice and lifelong learning.