Development of biomedical information systems: MSL concept of e-learning – pilot study results

Overview

Our needs brought us to the question of how to educate effectively a group of people with different levels of knowledge. The MSL (Multiple Step Learning) concept of e-learning is an effort to create supportive materials for our practical lessons that reflect the previous knowledge of our students and contain links for valuable resources. In the meantime, we have been running a pilot study to evaluate the quality of courses based on the concept. For the evaluation we have used modified eLSE methodology, which combines systematic inspection and user testing. The results show that our MSL courses are good quality and widely accepted by students. In the final step we would like to compare this concept to the previous way of teaching without e-learning. Thanks to the Mefanet project we are able to share our courses with other medical schools and test the concept with more users in the future.

Keywords:
e-learning, methodology, evaluation, MSL, adaptive systems

Introduction

At our department we are responsible for the biophysical lessons of the first year medical students. Medical biophysics is hardly called mainstream for our students and their approach differs a lot. However, the rapid development of new diagnostic machines, which are based mainly on physical principles, gives a reason for making this subject more attractive. Future physicians should be aware of the limits of the available diagnostic machinery.

Modern technologies change the classical way of education a lot. According to Bauer [2], teachers have lost their classical role of being the only information authority. However, they are not displaced. They get a new role, sometimes even more demanding and indispensible than before: the role of a guide on the sea of knowledge.

Thanks to the enormous variety and different quality of information sources nowadays, in a course it is more likely to have students who excel in certain topics but have totally insufficient knowledge of other topics. The problem is even more obvious if you deal with students from different countries. Thanks to the Internet and distant study programs, the internationalization of education is becoming more and more common. There are several ways to deal with it. A strictly defined level of knowledge for entering the course is one way. Another way, which anticipates a larger number of students, is to divide them into groups according their knowledge. However, if you are already facing the problem of a small group of students with very heterogeneous knowledge levels, you may find our solution valuable.

MSL concept

Basic idea, form 3SL to MSL

According to Šimon [10] ICT – Information and Communication Technologies – attract students to the educational process. ICT also individualize education and save expenses for paper materials. From our point of view, using ICT is not only about the individualization of the education process and saving funds. If we recognize the previous knowledge of students we can omit the corresponding further explanation; this brings us to saving time, which cannot be overlooked, either.

Saving time might be very interesting and attractive, especially for our first year medical students. For that reason we created the 3SL concept – Three Step Learning concept. This concept divides the topic into three levels with different details. The first level explains the topic completely but assumes deeper knowledge and some details are omitted. Basically all further levels fill the information gap of the previous one.

As it has been shown during testing in practice the 3SL concept was not universal enough. That is why we have decided not to limit the number of detail levels and the thought of the MSL (Multiple Step Learning) concept was conceived. The basic scheme of the MSL concept can be found in the picture.

Pilot study and implementation in Moodle

To run our courses we use Moodle LMS (Learning Management System) [11] which is the open source already used by other Czech medical faculties [3,7] and it is supported by a fast growing Czech community [8].

There is a fact often mentioned in literature that in order to use an e-learning system effectively, it is necessary to adequately prepare the teachers for teaching in the new environment. As far back as 2004, Prensky [9] aptly called students “digital natives” and teachers “digital immigrants”. The fact that teachers have general IT knowledge (such as text processing, Internet use and email use) says nothing about their ability to incorporate their regular teaching knowledge into on-line teaching.[5] That is why we use Moodle – at the moment mainly as an extension of classical teaching – to get our teachers used to it.

In the meantime, we have been running pilot blended MSL courses about principles of computer tomography, ECG and ultrasound imaging as support materials for our practical lessons. Apart from the lectures, the courses also consist of discussion forums, glossaries, links to sources of high-quality materials and, last but not least, practical XHTML webpage-style complex educational materials with the possibility of showing or hiding different detail levels.

The idea of MSL implementation to a Moodle course is quite simple. Moodle enables the creation of an interactive lecture with the information flow controlled by answers to different kinds of questions. A student who enrolls himself in the course will see fully summarized information as a first step. If he manages to understand this information and answers the questions correctly, the course and preparation for the practical lesson is over. If he or she does not manage to answer all the questions, a more detailed level focused on the problematic part is viewed. This can continue until all levels are viewed. If the student cannot understand even the most detailed level, the system tells him to consult a teacher during the practical lesson.

Evaluation

We based evaluation of our courses on the eLSE (e-Learning Systems Evaluation) method, which was developed by Italian researchers Ardito et al. [1]. The advantage of eLSE is the combination of a specific inspection technique with user testing and low costs. This technique allows inspectors who may not have wide experience in evaluating e-learning systems to perform accurate evaluations. It is based on the use of evaluation patterns, called Abstract Tasks (ATs), which precisely describe the activities to be performed during the inspection. ATs are formulated precisely by means of a pattern template which provides a consistent format and includes five items:

• AT Classification Code and Title - univocally identify the AT and its purpose
• Focus of Action - lists the application’s objects to be evaluated
• Intent - clarifies the specific goal of the AT
• Activity Description - describes in detail the activities to be performed during the AT application
• Output - describes the output of the fragment of the inspection the AT refers to.

ATs are usually defined by expert evaluators and they are usually divided into two classes: “Content learnability” and “Quality in use”.

Our modification is mainly about the implementation of the original idea to Moodle in order to have the evaluation done directly by users (students). For every AT we prepared a set of so called “rating questions” in the Feedback module of Moodle. Each of the rating questions may be answered either in a positive sense (encoded as 1 or yes) or in a negative sense (0 or no). Besides rating questions there are also survey questions. These questions do not have any rating value. The purpose of these questions is to give real feedback that shows how to improve the course in the future. At the end we have an evaluation questionnaire with precisely described tasks that are answered by a series of questions with defined answers. Obtained data can be easily exported from Moodle and statistically processed.

The result of the statistical processing is the decision whether the number of yes answers is greater than the conventional 60 %. If we translate it into the language statistics uses, we test a null hypothesis about the parameter π of the alternative distribution H0: π ≤ π0 against the alternative Ha: π > π0, where π represents the ratio of yes answers and π0 =0.6. The result of such calculations is the corresponding p-value. We may also calculate the power of the test 1-β for a given significance level α. If we choose the conventional significance level of the test α=0.05, we say the rejection of the null hypothesis is valid if the p-value is less than 0.05 and we say the acceptance of the null hypothesis is valid if the power of the test 1-β is conventionally greater than 0.80 i.a.w. Cohen [4] and Daly and Bourke [6]. If the power of the test 1-β is less than 0.80, we may estimate what the number of observations in our future research should be to satisfy our requirements for the power of the test. The calculations of the corresponding algorithm give us the size of the whole future sample.

Evaluation results

For the evaluation itself we used the following ATs:

• In class Quality in Use (QU) it is: system design, communication tools availability, self-assessment tools, users assessment
• In class Content Learnability (CL) it is: course structure, organization of information, feedback validity

Students were asked to do the evaluation after they went through the course and the practical lesson. Our main goal was to clarify if the information provided by the course is sufficient and clearly understandable. We processed the data in a way to get an overall score, p-value and the power of the test for the both classes of ATs and the course itself. However, the score with p-value and the power of the test can be obtained not only for each AT but also for each rating question.

In the table situated below, it can be seen that all the courses we created are significantly better than 60 % which was the level for a good quality course.

Comparison of years with and without e-learning support

The number of students that evaluated as improvement the classes supported by e-learning changed significantly. The table below is based on a questionnaire handed out at the end of each class. H0 hypothesis means that there is no change in evaluation of the classes with e-learning support implementation compared to the classes without e-learning.

We also evaluated the success of education with versus without e-learning in the period of years 2006-2010. We picked a set of test questions in specific topics. The same questions were used in years without e-learning and in the years in which e-learning was introduced. The success rate was tested using the test for binomial distribution. We found out that the use of our e-learning support increased the success rate in all the topics in which the MSL approach was used.

All our courses based on the MSL concept were found to be high-quality. Furthermore, we also have results which indicate that these courses made our practical lessons significantly more popular in comparison with the previous year. In spite of the fact that even our test success rate comparison confirmed the use of MSL approach as significantly better, we have to admit that preparation of educational materials in the form suitable for MSL presentation is a time-consuming activity and that is why this concept is less suitable for general use. According to our experience the MSL approach can be used with an advantage in case of a smaller number of students with different levels of knowledge. An alternative to this solution would be a module for real adaptive e-learning for Moodle. Unfortunately these modules are currently still under development.

Acknowledgement

“Supported by the programme PRVOUK P37/09”

MUDr. Tomáš Nosek, Ph.D.

Department of Medical biophysics

Charles University in Prague

Medical faculty in Hradec Kralove

Simkova 870, CZ-500 38 Hradec Kralove

E-mail: nosek@lfhk.cuni.cz

Phone: +420 495 816 255