How do you develop a cross-LMS usage comparison?

I recently posted about the need to develop an approach that allows for the simple and consistent comparison of usage and feature adoption between different Learning Management Systems (aka LMS, Virtual Learning Environments – VLEs – see What is an LMS?). That last post on the need didn’t really establish the need. The aim of this post is to explain the need and make some first steps in identifying how you might go about enabling this sort of comparison.

The main aim is to get my colleagues in this project thinking and writing about what they think we should and how we might do it.

What are you talking about?

Just to be clear, what I’m trying to get at is a simple method by which University X can compare how its staff and students are using its LMS with usage at University Y. The LMS at University Y might be different to that at University X. It might be the same.

They might find out that more students use discussion forums at University X. More courses at University Y might use quizzes. The could compare the number of times students visit course sites, or whether there is a correlation between contributions to a discussion forum and final grade.

Why?

The main reason is so that the university, its management, staff, students and stakeholders have some idea about how the system is being used. Especially in comparison with other universities or LMSes. This information could be used to guide decision making, identify areas for further investigation, as input into professional development programs or curriculum design projects, comparison and selection processes for a new LMS, and many other decisions.

There is a research project coming out of Portugal that has some additional questions that are somewhat related.

The main reason is that there currently appears to be no simple, effective method for comparing LMS usage between systems and institutions. The different assumptions, terms and models used by systems and institutions get in the way of appropriate comparisons.

How might it work?

At the moment, I am thinking that you need the following:

  • a model;
    An cross-platform representation of the data required to do the comparison. In the last post the model by Malikowski et al (2007) was mentioned. It’s a good start, but has doesn’t cover everything.

    As a first crack the model might include the following sets of information:

    • LMS usage data;
      Information about the visits, downloads, posts, replies, quiz attempts etc. This would have to be identified by tool because what you do with a file is different from a discussion forum, from a quiz etc.
    • course site data;
      For each course, how many files, is there a discussion forum, what discipline is the course, who are the staff, how many students etc.
    • student characteristics data;
      How were they studying, distance education, on-campus. How old were they?
  • a format;
    The model has to be in an electronic format that can be manipulated by software. The format would have to enable all the comparisons and analysis desired but maintain anonymity of the individuals and the courses.
  • conversion scripts; and
    i.e. an automated way to take institutional and LMS data stick it into the format. Conversion scripts are likely to be based around LMS and perhaps student records system. e.g. a Moodle conversion script could be used by all the institutions using Moodle.
  • comparison/analysis scripts/code.
    Whatever code/systems are required to take the information in the format and generate reports etc. that help inform decision making.

Format

I can hear some IT folk crying out for a data warehouse to be used as the format. The trouble is that there are different data warehouses and not all institution’s would have them. I believe you’d want to initially aim for a lowest common denominator, have the data in that and then allow further customisation if desired.

When it comes to the storage, manipulation and retrieval of this sort of data, I’m assuming that a relational database is the most appropriate lowest common denominator. This suggests that the initial “format” would be an SQL schema.

How would you do it?

There are two basic approaches to developing something like this:

  • big up front design; or
    Spend years analysing everything you might want to include, spend more time designing the perfect system and finally get it ready for use. Commonly used in most information technology projects and I personally think it’s only appropriate for a very small subset of projects.
  • agile/emergent development.
    Identify the smallest bit of meaningful work you can do. Do that in a way that is flexible and easy to change. Get people using it. Learn from both doing it and using it to inform the next iteration.

In our case, we’ve already done some work from two different systems for two different needs. I think discussion forums are shaping up as the next space we both need to look at, again for different reasons. So, my suggestion would be focus on discussion forums and try the following process:

  • literature review;
    Gather the literature and systems that have been written analysing discussion forums. Both L&T and external. Establish what data they require to perform their analysis.
  • systems analysis;
    Look at the various discussion forum systems we have access to and identify what data they store.
  • synthesize;
    Combine all the requirements from the first two steps into some meaningful collection.
  • peer review;
    If possible get people who know something to look at it.
  • design a database;
    Take the “model” and turn it into a “format”.
  • populate the database;
    Write some conversion scripts that will take data form the existing LMSes we’re examining and populate the database.
  • do some analysis;
    Draw on the literature review to identify the types of analysis/comparison that would be meaningful. Write scripts to perform that role.
  • reflect on what worked and repeat;
    Tweak the above on the basis of what we’ve learned.
  • publish;
    Get what we’ve done out in the literature/blogosphere for further comment and criticism.
  • attempt to gather partners.
    While we can compare two or three different LMS within the one institution. The next obvious step would be to work with some other institutions and see what insights they can share.

The knowledge and experience gained this for “discussion forums” could then be used to move onto other aspects.

What next?

We probably need to look at the following:

  • See if we can generate some outside interest.
  • Tweak the above ideas to get something usable.
  • Gather and share a bibliography of papers/work around analysing discussion forum participation.
  • Examine the discussion forum data/schema for Blackboard 6.3 and Webfuse.

That’s probably enough to be getting on about.

References

Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

Automating calculation of LMS/CMS/VLE feature usage – a project?

I’m in the midst of looking at the work of Malikowski et al in evaluating the usage of VLE features. The aim of this work is an attempt to provide information that can help those who help academics use VLEs. The following is an idea to address those problems and arrive at something that might be useful for cross-institutional comparisons.

Given the widespread adoption of the LMS/VLE, I’d be kind of surprised if someone hasn’t given some thought to what I’ve suggested, but I haven’t heard anything.

Do you know of a project that covers some of this?

Interested in engaging in something like this?

Their contribution

An important contribution they’ve made is to provide a useful framework for comparing feature usage between different systems and summarised the basic level of usage between the different parts of the framework. The framework is shown in the following image.

Malikowski Flow Chart

Limitations

However, there remain two important questions/problems:

  1. How do you generate the statistics to fill in the framework?
    Malikowski et al suggest that prior studies relied primarily on asking academics what they did with the LMS. They then point out that this approach is somewhat less than reliable. The adopt a better approach by visiting each course site and manually counting feature usage.

    This is not much of an improvement because of the workload involved but also the possibility of errors due to them missing usage. For example, the role in the LMS of the user visiting each course site may not be able to see everything. Alternatively, when they visit the site may change what they see e.g. an academic that deletes a particular function before term ends.

  2. What does it mean to adopt a feature?
    In Malikowski (2008) adoption is defined as using a feature more than the 25% percentile. This, I believe, is open to some problems as well.

Implications

Those limitations mean, that even with their framework, it is unlikely that a lot of organisations are going to engage in this sort of evalaution. It’s too difficult. This means less data can be compared between institutions and systems. This in turn limits reflection and knowledge.

Given the amount of money being spent on the LMS within higher education, it seems there is a need to address this problem.

One approach

The aims of the following suggestion are:

  • Automate the calculation of feature usage of LMS.
  • Enable comparison across different LMS.
  • Perhaps, include some external data.

One approach to this might be to use the model/framework from Malikowski et al as the basis for the design of a set of database tables that are LMS independent.

Then, as need arises, write a series of filters/wrappers that retrieve data from a specific LMS and inserts it into the “independent” database.

Write another series of scripts that generate useful information.

Work with a number of institutions to feed their data into the system to allow appropriate cross institutional/cross LMS comparisons.

Something I forgot – also work on defining some definition of adoption that improves upon those used by Malikowski.

Start small

We could start something like this at CQU. We have at least two historically used “LMS/VLEs” and one new one. Not to mention Col already having made
progress on specific aspects of the above.

The logical next step would be to expand to other institutions. Within Australia? ALTC?

Getting half-baked ideas out there: improving research and the academy

In a previous post examining one reason folk don’t take to e-learning I included the following quote from a book by Carolyn Marvin

the introduction of new media is a special historical occasion when patterns anchored in older media that have provided the stable currency for social exchange are reexamined, challenged, and defended.

In that previous post I applied this idea to e-learning. In this post I’d like to apply this idea to academic research.

Half-baked ideas

In this post Jon Udell talks about the dissonance between the nature of blogs, the narrative form he recommends for blogs and the practices of academics. In it he quotes an academic’s response to his ideas for writing blogs as

I wouldn’t want to publish a half-baked idea.

Jon closes the blog post with the following paragraph

That outcome left me wondering again about the tradeoffs between academia’s longer cycles and the blogosphere’s shorter ones. Granting that these are complementary modes, does blogging exemplify agile methods — advance in small increments, test continuously, release early and often — that academia could use more of? That’s my half-baked thought for today.

I think this perspective sums it up nicely. The patterns of use around the old/current media for academic research (conference and journal papers) are similar to heavyweight software development methodologies. They rely on a lot of up-front analysis and design to ensure that the solution is 100% okay. While the patterns of use of the blogosphere is very much more like that of agile development methods. Small changes, get it working, get it out and learn from that experience to inform the next small change.

Update: This post talks a bit more about Udell’s views in light of a talk he gave at an EDUCAUSE conference. There is a podcast of the presentation.

There are many other examples of this, just two include:

Essentially the standard practices associated with research projects in academia prevent many folk from engaging in getting the “half-baked ideas” out into the blogosphere. There are a number of reasons, but most come back to not looking like a fool. I’ve seen this many times with my colleagues wanting to spend vast amounts of time completing a blog post.

As a strong proponent and promoter of ateleological design processes, I’m interested in how this could be incorporated into research. Yesterday, in discussions with a colleague, I think we decided to give it a go.

What we’re doing and what is the problem?

For varying reasons, Col and I are involved, in different ways, with a project going under the title of the indicators project.. However, at the core of our interest is the question

How do you data mine/evaluate usage statistics from the logs and databases of a learning management system to draw useful conclusions about student learning, or the success or otherwise of these systems.

This is not a new set of questions. The data mining of such logs is quite a common practice and has a collection of approaches and publications. So, the questions for use become:

  • How can we contribute or do something different than what already exists?
  • How can we ensure that what we do is interesting and correct?
  • How do we effectively identify the limitations and holes underpinning existing work and our own work?

The traditional approach would be for us (or at least Col) to go away, read all the literature, do a lot of thinking and come up with some ideas that are tested. The drawback of this approach is that there is limited input from other people with different perspectives. A few friends and colleagues of Col’s might get involved during the process, however, most of the feedback comes at the end when he’s published (or trying to publish) the work.

This might be too late. Is there a way to get more feedback earlier? To implement Udell’s idea of release early and release often?

Safe-fail probes as a basis for research

The nature of the indicators project is that there will be a lot of exploration to see if there are interesting metrics/analyses that can be done on the logs to establish useful KPIs, measurements etc. Some will work, some won’t and some will be fundamentally flawed from a statistical, learning or some other perspective.

So rather than do all this “internally” I suggested to Col that we blog any and all of the indicators we try and then encourage a broad array of folk to examine and discuss what was found. Hopefully generate some input that will take the project in new and interesting directions.

Col’s already started this process with the latest post on his blog.

In thinking about this I can come up with at least two major problems to overcome:

  • How to encourage a sufficient number and diversity of people to read the blog posts and contribute?
    People are busy. Especially where we are. My initial suggestion is that it would be best if the people commenting on these posts included expertise in: statistics; instructional design (or associated areas); a couple of “coal-face” academics of varying backgrounds, approaches and disciplines; a senior manager or two; and some other researchers within this area. Not an easy group to get together!
  • How to enable that diversity of folk to understand what we’re doing and for us to understand what they’re getting at?
    By its nature this type of work draws on a range of different expertise. Each expert will bring a different set of perspectives and will typically assume everyone is aware of them. We won’t be. How do you keep all this at a level that everyone can effectively share their perspectives?

    For example, I’m not sure I fully understand all of the details of the couple of metrics Col has talked about in his recent post. This makes it very difficult to comment on the metrics and re-create them.

Overcoming these problems, in itself, is probably a worthwhile activity. It could establish a broader network of contacts that may prove useful in the longer term. It would also require that the people sharing perspectives on the indicators would gain experience in crafting their writing in a way that maximises understandability by others.

If we’re able to overcome these two problems it should produce a lot of discussion and ideas that contributes to new approaches to this type of work and also to publications.

Questions

Outstanding questions include:

  • What are the potential drawbacks of this idea?
    The main fear I guess of folk is that someone, not directly involved in the discussion, steals the ideas and publishes them unattributed and before we can publish. There’s probably a chance that we’ll also look like fools.
  • How do you attribute ideas and handle authorship of publications?
    If a bunch of folk contribute good ideas which we incorporate and then publish, should they be co-authors, simply referenced appropriately, or something else? Should it be a case by case basis with a lot of up-front discussion?
  • How should it be done?
    Should we simply post to our blogs and invite people to participate and comment on the blogs? Should we make use of some of the ideas Col has identified around learning networks? For example, agree on common tags for blog posts and del.icio.us etc. Provide a central point to bring all this together?

References

Lucas Introna. (1996) Notes on ateleological information systems development, Information Technology & People. 9(4): 20-39

The emperor has no clothes – why is the learning and teaching peformance fund naked

Vilhelm Pedersen illustration for Andersen's 'Emperor's New Clothes'

The Australian Federal Government has a Learning and Teaching Performance Fund (LTPF) that is meant to allocate money to Australian universities on the quality and/or improvement in their learning and teaching.

Based on what I know of this approach I think it is fundamentally broken. It’s probably that the “emperor has no clothes”. i.e. Australian universities know it is broken, but can’t point it out because they want to get the money.

The fund and how it works

According to this story in the Australian newspaper the fund allocated $AUD73 million this year. The administrative information for providers document outlines the process for 2009. The process has changed over the 3 years it has been run.

There are two data sources used by the fund:

  • Australian Graduate Survey; and
    All Australian University graduates get a survey in the months after they graduate which asks them two broad sets of questions: are they working and in what, and how satisfied were they with their study/university. The LTPF uses two sets of indicators from this survey
    • Student satisfaction indicators
      • Satisfaction with generic skills
      • Satisfaction with good teaching
      • Overall satisfaction
    • Outcome indicators
      • Full-time employment
      • Further full-time and part-time study
    • Higher education student collection.
      The statistics are used in the LTPF to examine progress rates amongst Bachelor students and the retention rate for the same students.

    The process used goes something like

    • An adjustment process is applied to the raw indicators data.
    • Each university gets a package describing details of the findings for their students.
    • The university provide a submission offering information that may explain some of the results.
    • An expert panel looks at the information and provides advice to the government, back to the institutions and generally ensures the process is effective.

    The trouble is that I think the majority of the data that is at the foundation of this process is less than reliable.

    Why is it broken

    A couple of weeks ago I published a post titled “Somethings that are broken with the evaluation of university teaching”. Essentially it is a collection of links pointing out that “level 1 smile sheets” (surveys that ask learners “were you satisfied”) have significant and well-known limitations in validity and value. In particular the following quote is from this article

    In some instances, there is not only a low correlation between Level I and subsequent levels of evaluation, but a negative one.

    From my perspective the course experience questionnaire (the bit of the Australian graduate survey that asks student satisfaction) essentially takes the “level 1 smile sheet” approach and applies it to a graduates entire university experience. I don’t see this move to a broader area of coverage (whole university experience, up from individual course/unit/subject) helping address the concerns about “level 1 smile sheets”. In fact, I see it getting much worse.

    The course experience is likely to cover at least 3 years experience. For some part-time students this might as much as 6, 9 or more years. Do we really believe that their experience over the last 6 to 12 months isn’t going to over shadow and be more in their mind than their previous experience?

    There are also problems with the graduate destination survey – the part of the survey that asks about what they are doing now. This article points out some of the limitations.

    To some extent having institutions comment on the data before the expert panel examines it might address some of this. But I don’t think that goes anyway towards addressing the significant limitations of this form of evaluation.

    Solutions

    So if it’s broken, what are the alternatives?

    I don’t know. It’s a difficult question and I don’t have the knowledge, experience or time to recommend a solution. Given the nature of this problem, I’m not even sure that there is a single correct solution.

    Based on what I know, some suggestions I think might be worth more consideration include:

    • Measure fit for context/purpose, not comparison.
      To some extent the LTPF process acknowledges that comparing the quality of learning and teaching across all of the diversity of the Australian higher education sector is extremely difficult. So why continue to try and do it. Why not focus on how well the learning and teaching is for the given context. Measure fit for purpose, not comparison against others. That said, any form of measurement has some potential downsides and negative outcomes.
    • Concentrate on improvement.
      This is somewhat similar to the last point. It’s also linked to one of my common sayings, “It’s not how bad you start, but how quickly you get better”. Rather than measure fit for context, measure and reward how much better the learning and teaching at a particular institution has become. Also some potential negative concequences.
    • Use other forms of evaluation/data.
      This and I’m sure many other places talk about additional forms of evaluation beyond level 1 smile sheets. Dave Snowden also has some interesting approaches to evaluation which might apply.

    In general, I would suggest that rather than wringing hands over how difficult, spending inordinate amounts of time reflecting on how hard it is and arguing over which of many options is the best, spending lots of money on consultants that will push their own barrow at the expense of any knowledge of the local context, leaping at the latest fad, or say it’s all too hard we can’t change the governments mind. I think it would suggest an organisation attempt a lot of safe-fail probes to investigate different potential solutions. Apply the lessons learned to improve evaluation within the institution and then promote it amongst the sector.