Plans for a AJET paper around the indicators project

The following is an attempt to make concrete some idea and tasks associated with writing a journal paper for the Australian Journal of Educational Technology (AJET) based on the ASCILITE’09 paper based on the work of the Indicators project. The paper will be co-authored by a group and the aim of this post is to start discussion about content of the paper and the tasks we need to do to get it written.

The purpose/abstract of the paper

One argument is that as a group of authors we should have a common sense of the purpose of the paper. We should have an elevator pitch about the paper, a 30 second spiel that summarises what we’re trying to achieve/show. Any improvements?

The use of Learning Management Systems (LMS) is almost ubiquitous within universities. However, few institutions are actively using the data generated by student and staff use of these systems to guide future work. This paper seeks to highlight:

  • Some limitations of existing attempts to leverage LMS usage data.
  • Illustrate knowledge about e-learning within universities that, because of these limitiations, appear not to be as certain or clear as existing literature or common-sense might suggest.
  • Indicate how the work within this initial paper can be used to design further work that can improve our understanding of what is happening with e-learning, why it is happening and how it can be improved.

This might become a useful tag line for the paper:

  • What is happening.
    Refers to the patterns we find in the usage data.
  • Why is it happening.
    Refers to additional research with different methods and theories that would be required to suggest and test reasons why these patterns might exist.
  • How to change practice.
    Refers to further research and insight that seeks to combine the what and why information with other theory to change how learning/teaching practice is changed.

Ideas for titles

The title of a paper is, from one perspective, a summary of the elevator pitch. It should attract the reader and inform them what they might find out. What follows are some initial ideas.

  • The indicators project: Improving the what, why and how of e-learning.
  • Student LMS activity, grades, and external factors: Identifying the need for cross-platform, cross-institutional and longitudinal analysis of LMS usage.
    This title is based on a narrowing down of the “what” to be shown in the paper to just the linkage between LMS activity, grades and external factors. i.e. exclude the feature adoption stuff.

The structure of the paper

At this stage, the structure of the paper for me is heavily based around the three aims of the paper outlined in the purpose/abstract section above. The basic structure would be:

  • General introduction.
    Explain the background/setting of the paper. i.e. widespread use of LMS as implementation for e-learning. Start of people doing academic analytics. The identification of some known patterns. The importance of knowing what is going on and how to improve it. The diversity of universities and how one size might not fit all. i.e. different universities might have different experiences.
  • Limitations of existing work.
    Seek to provide some sort of framework to understand the work that has gone before in terms of academic analytics and/or LMS usage log analysis.
  • Different perspectives on what is happening.
    Examine and explain how we’ve found patterns which seem to differ or provide alternative insights to what has already been found. The idea here is to establish at least 2 groupings of patterns that illustrate some differences between what we have found and what has been reported in the literature. Each of the groupings could have multiple patterns/findings but there would be some commonality. More on this below.
  • Further work.
    Argue that these differing findings suggest that there is value in further work that:
    • addresses the limitations identified in the 2nd section and,
      i.e. cross platform, cross-institutional and longitudinal.
    • also expands upon the findings found in the 3rd section.
      Moves from just examining the “what” into the “why” and “how”.
  • Conclusions.

The work to be done

Now time to identify the work that needs to be done.

Limitations of existing work

The basic aim of this section is to expand and formalise/abstract the knowledge/opinion about the existing literature around LMS usage analysis expressed in the ASCILITE paper.

The draft ASCILITE paper – prior to compaction due to space limitations – was working on a framework for understanding the literature based on the following dimensions:

  • # of institutions;
  • # of LMS;
  • time period;
  • Method.

Work to do:

  • Ensure that we have covered/gathered as much of the relevant literature as possible.
  • Examine that literature to see how it fits within the framework.
  • Identify from the literature any additional dimensions that might be useful.
  • Identify if there is any findings that support or contradict the findings we want to introduce in the next section.

Different perspectives on what is happening

This is the section in which we draw on the data from CQU to identify interesting and/or different patterns from what is found in the established literature. The biggest question I have about this section is, “What patterns/groupings do we use?”. The main alternatives I’m aware of are

  1. Exactly what we did in the ASCILITE paper.
  2. The slight modification we used in the ASCILITE presentation
  3. Drop the feature adoption stuff entirely and focus solely on the correlation between student activity, grade and external factors. Perhaps with the addition of some analysis from Webfuse courses.

Whichever way we go, we’ll need to:

  1. Identify and define the patterns we’re using
    e.g. The correlation between level of participation, the grade achieved and some external factors.
  2. Identify literature/references summarising what is currently known about the pattern.
    e.g. The correlation that suggests the greater the level of participation in a LMS, the better the grade.
  3. Identify ways in which the pattern can be measured (see below).
  4. Use the measure to examine the data at CQU.
    e.g. many of the graphs in the ASCILITE paper/presentation.
  5. Look for any differences between expected and what we see.
    e.g. LMS usage by HD students is less than others for AIC students and Super low courses
  6. Establish that the differences are statistically significant.
  7. Perhaps generate some initial suggestions why this might be the case.

The patterns

The patterns we’ve been using so far seem to fit into one of two categories. Each of these categories has a definition of the pattern and how we’re actually measuring it. This is also an area of difference, i.e. there could be different ways of measuring.

The patterns we’ve used so far:

  1. % of courses that have adopted different LMS features.
    This is the comparison of feature usage between Blackboard and Webfuse during the period of interest. It shows that different systems and different assumptions do modify outcomes.

    • Current measurement – Malikowski et al
    • Alternative measurement – Rankine et al (2009)
  2. The link between LMS activity, student grades and various external factors.
    We’re currently measuring this by
    • # of hits/visits on course site and discussion forum.
    • # of posts and replies to discussion forum.

    The external factors we’ve used in papers and presentations are:

    • mode of delivery: flex, AIC, CQ.
    • Level of staff participation.
    • Different staff academic background.
    • Impact of input from instructional designer.
    • Age for FLEX students.

    I think there are a range of alternative measures we could use, need to think more about these.

We need to come to some consensus about the patterns we should use.

Statistical analysis

In the meantime, however, I think that we will end up using the activity/grades correlation at least for:

  • Mode of delivery.
  • Level of staff participation.
  • Age for FLEX students.

I would suggest that having the statistical analysis done and written up for these three would be a good first step. At least while we talk about the other stuff.

Further work

In terms of the further work section of the paper I think we need to:

  • Summarise the recommendations from the literature.
  • Identify where we might disagree and why.
  • Identify what we think should be done.

In terms of further work, my suggestions would be:

  • What
    • Testing the existing patterns in cross-platform, cross-institutional and longitudinal ways.
    • Establishing and testing alternate and additional patterns. e.g. the SPAN work
    • Establishing and testing alternate measurements for these patterns
    • Testing and developing alternate methods to make these patterns and enable different institutions to use them.
    • Identifying theories that would suggest other patterns which might be useful.
    • How to make these patterns available to front-line teaching staff and academics.
  • Why
    • Lots of work seeking to explain the differences in patterns found.
    • Identifying theories that help explain the patterns.
  • How
    • How to make these patterns available to staff and students in a way that encourages and enables improvement.
    • How to encourage management not to use these patterns as a stick.

Work to do

What follows is a list of tasks, by no means complete:

  • Everyone
    • Read and comment on the elevator pitch. Does it sound right? Can it be made better? Is there a different approach?
    • Does the proposed structure work? Should there be something more?
    • Suggestions for a title.
    • Thoughts about what patterns we use in the paper.
  • Stats guy
    • Have done the analysis on the “mode of delivery”, “level of staff participation” and “Age for flex students” patterns.
    • Be able to help us write a blog post that summarises/explains the analysis for each pattern in a way that would be suitable for the journal paper.
  • Non-stats guys
    • Combine existing literature around LMS usage analysis into a single bibliography, actively try and fill the holes.
    • Analyse the literature to help develop the framework for understanding and comparing the different approaches.
    • Pull out any interesting patterns, measures or findings that either support or contradict what we’ve found.
    • Writing
      • Introduction
      • Limitations of existing work
      • Future work.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s