Evaluations of online learning activities based on LMS logs

The following is a summary and some thoughts on Lam et al (2012). The abstract from the chapter is

Effective record-keeping, and extraction and interpretation of activity logs recorded in learning management systems (LMS), can reveal valuable iriformation to facilitate eLearning design, development and support. In universities with centralized Web-based teaching and learning systems, monitoring the logs can be accomplished because most LMS have inbuilt mechanisms to track and record a certain amount of information about online activities. Starting in 2006, we began to examine the logs of eLearning activities in LMS maintained centrally in our University (The Chinese University of Hong Kong) in order to provide a relatively easy method for the evaluation of the richness of eLearning resources and interactions. In this chapter; we. 1) explain how the system works; 2) use empirical evidence recorded from 2007 to 2010 to show how the data can be analyzed; and 3) discuss how the more detailed understanding of online activities have informed decisions in our University.

It’s a chapter in an IGI book, which means gaining access was not easy.

Thoughts

Well it is an example of longitudinal examination of LMS logs from one university – 2007, 2008 and 2009. Detail some of the considerations in doing a cross-LMS comparison. Find some interesting outcomes (e.g. use of LMS functionality in courses drop significantly as time went by) but without further research unclear what the reasons are.

It remains a surprise to me that the two Universities I’m most familiar with, don’t have something like this in place already to guide what they are doing in support.

Introduction

Starts with a definition of an LMS.

Moves onto discussion of log analysis. Including some older references.

Difficulties of doing institutional analysis as web-log tracking tools aimed at the individual teacher (and we know how effective those are). Made even more difficult with the version of WebCT they were using. Leading to a need to investigate the database and develop their own software.

Their focus more institutional, hence not a close monitoring of student activity, but does include both LMS.

Earlier work reported on data interpretation, focus here on automation of interpretation and reporting. Earlier work suggests log data can provide information on

  1. Popularity – yes/no indication per course whether any eLearning activities are recorded in the logs.
  2. Nature of functions/strategies – what facilities are used.
  3. Engagement of teachers and students – how involved folk are in the activities.

Frame these three as steps. 1) popularity indicates whether there is a course website, 2) nature reveals what is there, and 3) engagement shows how it is used/engaged with.

Claimed that the data to some extent fits the requirements for a naturalistic research paradigm. Recognises the need for comprehensive evaluation studies to consider other forms of evidence and sources.

Lists benefits of the log data approach

  • relative ease of access;
  • non-intrusive
  • Repeated measures enable longitudinal comparisons
  • with automation can enable an institutional system .

Drawbacks

  • monitors only use of the LMS
  • bias on quantity rather than quality
  • the activities are fairly abstract (e.g. discussion forum, content file etc) and not institutional specific (e.g. is the content file a course outline?)
  • The picture from logs is partial

How the system works

Starts with the measures used to refine and standardize the data. Especially due to the two LMS.

  • Not al websites on the LMS are active (made available to students).
  • Other considerations for “active” other strategies – had at least one student
    • accessed any forum.
    • attempted any quiz.
    • submitted an assignment.
    • download a content file.
  • Only included classes with at least 10 students or more.

Describes the actual system which appears to have been a web-based report/query system. Choose various variables and generate comparisons against popularity, nature and engagement. Levels of analysis include

  • institution-wide overview of eLearnign activities
  • differentiate faculty or department level eLearnign practices
  • popularity of the different LMS
  • How students and teachers are engaged in various activities

Some sample findings

These samples are provided to be illustrative. Usually at faculty level and being reported across the 3 years (2007-2009)

  • Popularity of LMS

    More courses are using the web. Both LMS percentage increased. Moodle increased quickly from introduction in 2007.

  • Comparison of two functions

    Main use was for content delivery.

    • 90% of active websites contained content (reducing from 96.6% (1432) in 2007 to 91.9% (1891) in 2009).
    • 23.8% (2007), 7.1% (2008), 6.3% (2009) had active quizzes.
    • 21.5%, 20.9% and 14.8% had discussions.
    • 27.6, 24 and 14.6% used online assignment submission.

    No discussion/explanation about why the reduction in percentage.

  • Engagement in four areas

    90% of students access sites. 30% accessed the LMS more than 20 times during the year. Aside: an incredibly low figure, perhaps the on-campus only factor?

    Only 1995 students out of 15,000 wrote anyting in a forum. Most only wrote one or two. Same with assignment submission.

    Different with quizzes. 4742 used quizzes, one-third attempt 11-20 quizzes. 10% made 20 or more attempts.

  • Use of LMS in four faculties

    Faculties used the LMS differently. Experience not the same across disciplines.

    All faculties increased course websites. But overall percentage of courses with websites varied between faculties.

  • Use of three functions in four faculties

    All used content. Different use of other functions was observed. One faculty had 50% of courses with online forums in 2007 dropping to 5% in 2009. Similar observations made.

    Some explanations given arising from personal communications. e.g. new faculty started strong with e-learning but as lots of new staff arrived and teaching loads increased e-learning suffered.

Refining our elearning strategies

Makes claims about the value of this type of analysis

Some more general discussion. e.g. the content focus with some references about the limitations of technologies to change learning and teaching.

Highlights the content centric nature of the LMS Aside: I wonder how that gels with the Moodle socio-constructivist design philosophy?

Bringing up the modern educational trend – learner centered – teacher as facilitator etc.

findings thus suggest that institutional eLeam- ing support should not merely focus on having a web presence in courses or using the Web for courseware delivery. Attention also needs to be on the diffusion and sustained use of interactive online learning activities

Raises concerns about sustainability of e-learning, given the descrease in feature usage observed. Uncertain about what the cause is. Suggest it might be an LMS problem given observations of teachers using Web 2.0 strategies. Raise some questions about percieved usefulness etc.

MIght also be the staff rejecting online. Does mention that the university is face-to-face.

Mentions difficulties with engaging staff and the institution reviewing its support measures.

References

Lam, P., Lo, J., Lee, J., & Mcnaught, C. (2012). Evaluations of Online Learning Activities Based on LMS Logs. In R. Babo & A. Azevedo (Eds.), Higher Education Institutions and Learning Management Systems: Adoption and Standardization (pp. 75–93). Hershey, PA: IGI Global.

3 thoughts on “Evaluations of online learning activities based on LMS logs

  1. Pingback: Examining diffusion and sustainability of e-learning strategies thorugh weblog data | The Weblog of (a) David Jones

  2. Pingback: Evaluations of online learning activities based...

  3. Pingback: Challenges in employing complex e-learning strategies in campus-based universities | The Weblog of (a) David Jones

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s