The Indicators project is an attempt to enable the examination, analysis and comparison of LMS usage across time, systems and institutions. The project is nothing new. Projects around academic analytics have been around for a while and I and others at my current institution have been talking about using the collective data in systems logs to inform the practice of L&T at universities for a long time.

Given the long term interest and the need to do this, why aren’t more universities doing it? What is getting in the way? This post is the start of a list of factors that I have gotten an inkling of through the literature, personal experience and talking with others. What would you add?

(PS. I’m trying to follow advice in this post on improving a blog post)

Nature of the technology: Limitations of the LMS

Currently, for most universities, the learning management system (LMS – Blackboard, Moodle etc) are the main environment for L&T. For good or bad, this is a major source of data. Some of the barriers are inherent to the nature of the LMS. These include:

• Minor variations in naming and approach.
  Even though there are more similarities than differences between different LMS (Black et al, 2007), those differences are enough to make comparisons between LMS somewhat difficult. There needs to be work like that carried out by Malikowski et al (2007) to enable meaningful comparisons.
• The LMS focus on individual courses.
  An LMS is focused on enabling individual academics create and maintain course sites. This means the features, including reporting, are targeted at the course level.
• The LMS doesn’t encompass all data.
  To really understand the impact of LMS activity you need to be able to match student activity with student performance (i.e. their grades). Typically, student grade information is not available within the LMS. If you all rely on is the LMS data, you can’t access this and other information (e.g. demographic information about students or staff).
• Advice to clear system logs.
  The versioni of Blackboard my institution used to run came with some advice to purge the activity accumulator. One of the major tables/systems of the LMS that recorded what was being done with Blackboard. Purging it means destorying the data. Unless the IT folk involved stored the data first, you lost it.
• The general poor quality of the reporting/visualisation features of an LMS.
  From Dawson and McWilliam (2008) “the current LMS present poor data aggregation and similarly poor visualisation tools in terms of assisting staff in understanding the recorded patterns of student learning behaviour.”

Technical furphies

It’s been my experience that when technical people don’t really want to something, for whatever reason, they come up with a furphy to explain why it can’t be done. The best of these furphies have an element of truth in order to make them plausible. However, if you know a bit more about technology, there are usually simple solutions. The ones I’m aware of include:

• It will overload the live database.
  The idea here is that trawling the LMS database to generate analysis of how it is being used will raise “concerns with system load for querying the database in order to extract the required data” (Dawson & McWilliam, 2008). i.e. the users of the LMS will suffer from poor performance. The problem with this furphy is that any IT department worth its salt will already have a copy of the database that is being regularly updated that it uses for development and testing. If they don’t, they’re not doing their job properly. That test database is not directly used by users and can generally be used without performance concerns. In addition, with any of the “enterprise” database systems, it shouldn’t be difficult to create another copy of the database, if they don’t want the analytics playing with the dev database.
• We don’t have enough resources.
  This is perhaps a superset of the previous point. It usually means that we don’t have the database adminstrators, developers or hardware resources available to support the project. Providing access to an existing dev database on an “enterprise database” should take seconds – at least if the processes and automation used by the IT department is any good.

  In defence of the IT folk and also to make clear that these are not universals. In some cases, they don’t have the resources. Many institutions don’t put enough resourcing into IT to achieve what they want. In these cases, IT are probably better off collaborating with people who want to do things as a better way to generate support to get the additional resources.

• The database is to complicated to understand.
  Some of the folk in L&T working in analytics don’t have technical backgrounds and must rely on IT. As another version of the previous points there is the argument that the databases are too complicated to easily get useful data from them.
• It’s not us, it’s the system owner.
  We’d love to give you access, but first you have to talk to Y as they are the system owner. We’re pretty sure that they are not keen on the idea. Not sure how you might contact them. Just a bit of passive resistance.

Organisational barriers

• Ownership of the data.
  Some teaching staff see all information about their course in the LMS as belonging to them. Some faculties believe the same for their faculty courses. IT may see ownership of the database as belonging to them. The institution may not have any firm written policy on this and simply rely on who speaks most strongly. This problem often connected with the following.
• Mismatch with system owner requirements.
  For example, the organisational owner of student records data at our institution has been the student administration section. Folk responsible for getting students enrolled, results processed and accepting money. These folk are not mostly focused on improving learning and teaching. So may not see the rationale for analytics or other tasks.
• Privacy concerns.
  Fears that information about individual students or even staff, will be made public and or misused — leading to the next point.
• Plans and fears of using the data as a stick.
  It’s not hard to see some management using reports from the LMS to identify “bad” academics and punish them. This may be achieved simply through the creation of KPIs and standards for course delivery that assume that it’s possible to make such universal statements and ignore the inherent variability in university learning and teaching. This leads to the next one….
• Task corruption and Goodhart’s law.
  If you set a particular measure as a performance indicator/target (e.g. the presence of a discussion forum or the number of contributions to the discussion forum by the staff member) you will get people achieving that target. However, in some/many cases they will be using task corruption to achieve it.

Solutions

Many people complain that I point out problems with out pointing out solutions, hence the inclusion of this section

The solutions we’ve used, mostly by luck than forethought, have included:

• Historical responsibilities.
  At one stage, members of our project were system owners or designers of at least two of the LMS used by our institution. This meant that we have access to the databases for these systems. Though it didn’t help with the LMS we aren’t responsible for.
• Accidents.
  As mentioned above, advice from Blackboard is that you regularly clear/rotate the activity accumulator. For some reason, our IT folk never did this. So all that data, going back to 2005, was available.
• Technical knowledge.
  At least two of our project members have some technical knowledge. When faced with technical furphies we could point out alternate representations.
• Organisational knowledge.
  I’ve been at my current institution for nearly 20 years (god that’s sad). For good and bad, I know a fair bit about the organisation and the people within it. Often it is who you know, not what. For example, we got access to student records data over 10 years ago because I knew the system owner of the student data regularly had lunch at a certain place. I just happened to be there to ask nicely to have access to the data.
• Personal connections.
  Both internally and externally project members know a broad cross-section of people who can help provide alternate representations of stories that have been told. Representations that often unlock doors.
• Organisational power.
  This is perhaps the most useful one. If you are doing work that is seen to be useful or important by one person in power, or lots of people throughout the organisation, that can often provide the necessary power to override/workaround objections.
• Workarounds.
  Often you don’t have to ask permission. Often there are technical and social solutions to access that can work around barriers. For example, Bakharia et al (2009) talk about an approach to analyse course discussion forums that use Greasemonkey to avoid the need to access the LMS database.

Of course, in a perfect world, all members of the organisation would be warm and fuzzy folk who are only too happy to collaborate with their colleagues to achieve important outcomes for the organisation. None of them would ever be advised not to provide assistance or support to individuals within the same organisation.

References

Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.

Black, E., D. Beck, et al. (2007). “The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments.” Tech Trends 51(2): 35-39.

Dawson, S. and E. McWilliam (2008). Investigating the application of IT generated data as an indicators of learning and teaching performance. Melbourne, Australian Learning and Teaching Council: 45.

Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

The Excellence in Research for Australia (ERA) Initiative combined with various local institutional and personal factors has increased the interest of some colleagues and I in biting the bullet and entering the next stage of academic development. i.e. the preparation, submission and hopefully successful receipt of applications for external funding. This post is the start of some reflections and thinking about how and what to do.

I am starting from the assumption that it is now an institutional requirement that we undertake this task and be successful at it. There are a range of questions about the validity of aspects of this task and the thinking that may be displayed below. In part, this is purely pragmatic and, hopefully, will in part be mitigated by trying to knock the rough edges off that pragmatism by various other strategies.

This post is in part sparked by the announcement of a grant writing workshop being held at our institution next Tuesday. The announcement included some questions that we’re expected to have considered prior to the workshop. I’m using those questions as the structure for the rest of this post.

What is the research question/idea?

The basic idea/project is based around the Indicators Project. We’ve developed a tag line for the project, which summarises the basic aim

Enabling comparisons of LMS usage across institutions, platforms and time

i.e. we want to look at the usage – via system logs – of learning management systems (LMS) within universities. In particular, we want to look at, understand and compare the usage of LMS between different institutions, different LMS and do so longitudinally.

The question is how do we translate that into something that is important and attractive to the folk evaluating research proposals and is something we can actually do.

The answer to that, to some extent, is going to lie within the aims of the particular research grants and a range of other factors. For this project there are probably two very different types of grants:

1. ALTC (teaching and learning) grants; and
   There are also some internal L&T grants, this will focus on the ALTC grants.
2. Research grants.
   In our context, the ARC national competitive grants program is the nirvana. The type of grant you need to get to say you’ve arrived as a researcher. There are also equivalent grants internally at most universities within Australia. These are the stepping stone to the externally competitive grants.

This project could connect with both sides. The ALTC grants could be seen as helping with the “development” side of the project in terms of developing technology/knowledge support institutions to look at their LMS logs and those of others. The research grants are more connected with generating theoretical knowledge. For this project, identifying the ALTC grant maybe a bit easier.

ALTC grants

This project is probably best suited to a ALTC competitive grant. However, on the negative side there ALTC did fund an analytics related project this year. It built on a 2007 project. Any related project would have to work hard to demonstrate a difference between that project but still have some sense of connection or accumulation. The details of these grants are available from the ALTC site, so we can find out what they are aiming to do and see what we can do differently. Will be interesting to see if the idea could get up.

Can this idea be turned into a priority project – “academic standards, assessment practices and reporting”, “curriculum renewal” – are two of the priorities.

Priority grant applications are due April. Competitive due June.

ARC grants

As per the ARC site these are scheme-based (basically discovery and linkage) and across 6 inter-disciplinary groups

• Biological Sciences and Biotechnology
• Engineering and Environmental Science
• Humanities and Creative Arts
• Mathematics, Information and Communication Sciences
• Physics, Chemistry and Earth Sciences
• Social, Behavioural and Economic Sciences

I’m somewhat doubtful we can aim for these. Might be a step too far. The indicators, at initial glance, probably doesn’t fit nicely within those disciplines in order to be seen as something important. Think we’re going to have to identify the discipline and then get to know what things are important to the discipline. This page has lists of previously successful applications grouped by a number of ways including by field.

Discovery objectives are:

• support excellent fundamental research by individuals and teams
• enhance the scale and focus of research in the National Research Priorities
• assist researchers to undertake their research in conditions most conducive to achieving best results
• expand Australia’s knowledge base and research capability
• foster the international competitiveness of Australian research
• encourage research training in high-quality research environments
• enhance international collaboration in research

Linkage grants might be the way to go

The Linkage Projects scheme supports collaborative research and development projects between higher education organisations and other organisations, including within industry, to enable the application of advanced knowledge to problems. Typically, research projects funded under the scheme involve risk.

The idea might be to link with some of the LMS vendors.

Discovery applications look like closing March 2010. Linkage in May 2010.

This seems to be the page for seeing examples and getting some ideas.

Why is it important? What is significant and innovative in the study?

The first question is somewhat easy and has probably been made by the existing funded ALTC grants identified above. The points are:

• LMS are almost, if not, ubiquitous within higher education. Everyone has one.
• Few if any institutions are using the data about the use of these systems to drive decisions at any level.
• Drawing on this information can be used to improve the quality of learning and teaching at a number of different levels.
• More broadly, L&T is important in the knowledge economy etc.

Work will need to be done to connect this with priorities set by the funding agencies and/or the government.

The second question is a little more difficult given that there is existing work in the field (ALTC grants) and the question about how this gets linked under disciplines within ARC grants. More work needed here.

Do I need a research team on this project and who should be a team member? Why?

Some reasons for team members:

• Additional areas of expertise – e.g. statistics, educational research methods.
• Prestige and track record. i.e. this is important, I believe, for ARC grants. At the moment, I don’t think the existing indicators project members have the appropriate level.
• Other organisations. e.g. folk at other institutions to enable use of that institution as a site or perhaps someone who works for an LMS vendor or similar for the linkage grant angle.

Will need to look at the allowed size of teams.

What will be the key outcomes/contributions to the area/discipline?

This is related to the question of how it’s different from other work. Most broadly increased information to aid decision making. Perhaps the project needs to include an aim to identify, develop, or research how that decision making can be improved. i.e. not enough just to provide the information, need help on designing interventions and how the data is used to improve practice.

Perhaps guidelines/theory/knowledge that guides e-learning/learning and teaching becomes an outcome?

Misc other things

There was a Discovery project funded from 2009 title “Competing on Business Analytics” awarded to some IS academic from Uni Melbourne. Might have some useful insights. More info here and here. The latter provides some potential useful ideas.

Context

I’m currently sitting in the QANTAS club at Canberra airport waiting to return home after a week at ANU working on the PhD (being done through ANU). I decided to read a copy of CIO magazine while having brunch. In doing so I came across this article (Rodgers, 2009) title “Mind your own Business Intelligence.

This caught my eye because it mentions business intelligence. Business intelligence is very close to, for some it encompasses, the kind of work we’re starting in the Indicators project.

Business intelligence has dropped from the top 10

What interested me was this paragraph from the article

Perhaps even more surprising, however, was the fact that business intelligence dropped out of the top 10 items on CIOs’ agendas next year. Getting the right information to the right people at the right time for the right cost is what it takes to succeed in today’s business environment. Perhaps business intelligence sliding out of the top 10 is an indication of just how difficult BI is to achieve. BI is a moving target; it’s something that senior IT execs must constantly monitor and review to ensure that their organisation is getting the right information to key employees.

I find this interesting on two fronts:

1. CIOs are losing interest in business intelligence.
2. The slight touch of PEBKAC being implied as one of the problems.

PEBKAC

PEBKAC is an acronym devised by IT professionals as a code word for user error. i.e. the stupid user has made another mistake. PEBKAC expands out to Problem Exists Between Keyboard And Chair – i.e. the user. There is a tendency for IT folk to blame the user, instead of the technology.

The Wikipedia page on PEBKAC provides an important and interesting alternate perspective

Interface designers dismiss the blame on users for such trivial errors, arguing that a system that induces users to make mistakes is a badly designed one. By not taking human factors into consideration, its specification is incomplete and can make false or untested assumptions about the experience, knowledge and natural limits of their expected audience. Since the design lacks a major source of requirements, the resulting system will not be tailored to its purpose. The misunderstanding of the system that leads to the error is fault of the designer, not the user.

i.e. the problem is that system is designed for people to use correctly.

From this perspective, perhaps the problem with business intelligence isn’t that IT execs haven’t been constantly monitoring and reviewing the use of business intelligence to ensure that “their organisation is getting the right information to the key employees”.

Perhaps the technology the processes IT are using around business intelligence are broken. Perhaps they have not taken “human factors into consideration, its specification is incomplete and can make false or untested assumptions about the experience, knowledge and natural limits of their expected audience”.

I agree that business intelligence is really difficult, I believe most of its problems is that the tools and processes used to implement BI within organisations is broken.

The history of technology mediated learning

In an earlier post I argued that there is a highly visible hype cycle around e-learning/technology mediated learning. I think the same hype cycle exists in broader technology. This hype cycle is more closely aligned with Birnbaum than that of Gartner.

The four (grown from three) phases in the cycle I use are:

1. Technological spark – some new technology sparks interest, or enables new capabilities, solves an existing problem.
2. Growing revolution – a collection of folk find the spark important and a “club” grows around the technology building it up as the saviour of all.
3. Minimal impact – oops, it didn’t really make all that much difference, not many folk used it. Oh well.
4. Resolution of dissonance – the smart folk who pushed the revolution have to explain away why they were wrong. They can’t blame themselves. So they blame the users.

I think business intelligence may be getting into stage 3. Just like LMSes. But with LMSes, everyone is now going open source. The next fad.

References

Birnbaum, R. (2000). Management Fads in Higher Education: Where They Come From, What They Do, Why They Fail. San Francisco, Jossey-Bass.

Rodgers, M. (2009). Mind your own Business Intelligence. CIO. Summer 2009/2010: 4.

A couple of colleagues and I are trying to kickstart a little thing we call the Indicators project. We’ve developed a “tag line” for the project which sums up the core of the project.

Enabling comparisons of LMS usage across institutions, platforms and time

The project is seeking to enable different people at different institutions to analyse what is being done with their institutions learning management system (LMS, VLE, CMS) and compare and contrast it with what is happening at different institutions with different LMS.

To some extent this project is about improving the quality of the data available to decision makers (which we define to include students, teaching staff, support staff and management). In part this is about address the problem identified by David Wiley

The data that we, educators, gather and utilize is all but garbage.

But it’s not just about the data. While the data might be useful, it’s only going to be as useful as the people who are seeing it, using it and talking about it. David Warlick makes this point about what’s happening in schools at the moment

not to mention that the only people who can make much use of it are the data dudes that school systems have been hiring over the past few years.

And then this morning George Siemens tweeted the following

Collecting data less valuable that connecting people” http://bit.ly/3SMJCT agree?

If it’s an either/or question, then I agree. But with the indicators project I see this as a both/and question. For me, the indicators project is/should be collecting data in order to connect people.

What follows is an attempt to map out an example.

The link between LMS activity and grades

There is an established pattern within the literature around data mining LMS usage logs. That pattern is essentially

the higher the grade, the greater the usage of the LMS

The order is reversible as I don’t think anyone has firmly established a causal link, it’s just a pattern. My belief (yet to be tested) is that is probably, mostly good students get good grades and do everything they can do to get good grades, including using the LMS.

With our early work on the indicators project we have found some evidence of this pattern. See the two following graphs (click on them to see bigger versions).

The X axis in both graphs is student final grade at our current institution. From best to worst the grades are high distinction (HD), distinction (D), credit (C), pass (P), and fail (F).

In the first graph the Y axis is the average number of hits on either the course website or the course discussion forum. Hopefully you can see the pattern, students with better grades average a higher number of hits.

In the next graph, the Y axis is the average number of posts (starting a discussion thread) and the average number of replies (responding to an existing discussion thread) in the course discussion forum. So far, the number of replies is always greater than the number of posts. As you can see, the pattern is still there, but it is somewhat less evident for replies.

Importance of staff participation

Fresen (2007) identified the level of interaction or facilitation by teaching staff as a critical success factor for web-supported learning. We though we would test this out using the data from the project by dividing courses up into categories based on the level of staff participation.

The previous two graphs are actually for the 678 courses (the high staff particiaption courses) for which teaching staff had greater than 3000 hits on the course website during the term. The following two graphs show the same data, but for the super-low staff participation courses (n=849). A super-low course is one where teaching staff had less than 100 hits on the course website during term.

What do you notice about the pattern between grade and LMS usage?

First, the hits on the course site and the course discussion forum

Now, the average number of posts and replies in the course discussion forum

For me, the pattern is not there. The HD students appear to have decided there’s no value on the course website and decided they need to rely upon themselves. They’ve still been able to get a HD in spite of the super low staff participation. More work needs to be done.

I’m also interested in what the students in these super low courses might be talking about and what networks they are forming. The SNAPP tool/work at Wollongong might be useful here.

How to bring people together

My fear is that this type of finding will be used to “bring people together” in a way that is liable to be more destructive than anything. i.e. something like this:

• The data mining dweebs (I do recognise that this probably includes my colleagues and I) will bring it to the attention of university management.
  After all, at least at my institution it’s increasingly management that have access to the dashboards, not the academic staff.
• The data mining dweebs and management will tell stories about these recalcitrant “super-low” academics and their silliness.
• A policy will be formulated, probably as part of “minimum standards” (aka maximum requirements), that academics must average at least X (probably 3000 or more) hits on their course website in a term.
• As with any such approach task corruption will reign supreme.

While the indicators project is a research project focused on trying to generate some data, we also have to give some thought and be vocal about how the data could be used appropriately. Here are some initial thoughts on some steps that might help:

• Make it visible.
  To some extent making this information visible will get people talking. But that visibility can’t be limited to management or even teaching staff. All participants need to be able to see it. We need to give some thought about how to do this.
• Make it collaborative.
  If we can encourage as many people as possible to be interested in examining this data, thinking about it and working on ways to harness it to improve practice, then perhaps we can move away from the blame game.
• Be vocal and critical about the blame game.
  While publicising the project and the resulting data, we need to continuously, loudly and effectively criticise the silliness of the “blame game”/policy approach to responding to the findings.
• Emphasise the incompleteness and limitation of the data.
  The type of indicators/data we gather through the LMS is limited and from some perspectives flawed. An average doesn’t mean a great deal. You can’t make decisions with a great deal of certainty solely on this data. You need to dig deeper, use other methods and look closer at the specifics to get a picture of the real diversity in approaches. There may be some cases where a super-low staff participation approach makes a lot of sense.

References

Fresen, J. (2007). A taxonomy of factors to promote quality web-supported learning. International Journal on E-Learning, 6(3), 351-362.

This post is an attempt to capture some adhoc, over night thoughts about how the indicators project might move forward.

Context

Currently the indicators project is an emerging research project at CQUniversity. There are currently three researchers involved and we’re all fairly new to this type of project. I’d characterise the project at being at the stage where we’ve laid a fair bit of the ground work, done some initial work, identified some interesting holes in the literature around analytics/LMS evaluation and made the observation that there is a lot of different ways to go.

The basic aim is to turn the data gathered in Learning Mangement Systems (LMS, aka CMS, VLEs) usage logs into something useful that can help students, teaching staff, support staff, management and researchers using/interested in e-learning make sense of what is going on so they can do something useful. We’re particularly interested in doing this in a way that enables comparisons between different institutions and different LMS.

The process

A traditional approach to this problem would be big up front design (BUFD). The idea is that we spend – or at least report that we spent – lots of time in analysis of the data, requirements and the literature before designing the complete solution. The assumption is that, like gods, we can learn everything we will ever need to know during the analysis phase and that implementation is just a straight forward translation process.

Frankly, I think that approach works only in the most simplistic of cases, and generally not even then because people are far from gods. The indicators project is a research project. We’re aiming to learn new things.

For me this means that we have to adopt a more emergent, agile or ateleological approach. Lots of small steps where we are learning through doing something meaningful.

Release small patterns, release often

So, rather than attempt to design a complete LMS and institutional independent data schema and associated scripts to leverage that data, lets start small, focus on one or two interesting aspects, take them through to something final and then reflect. i.e. focus on a depth first approach, rather than a breadth first.

As part of this we should take the release early, release often approach. Going breadth first is going to take some time. Depth first we should be able to have something useful that we can release and share. That something will/should also be fairly easy for someone else to experiment with. This will be important if we want to encourage other folk, from other institutions to participate.

We should also aim to build on what we have already done and also build on what other people have done. I think that the impact on LMS usage by various external factors might be a good fit.

External factors and LMS usage

First, this is a line of work in which others published. Malikowski, Thompson & Theis (2006) investigate what effect class size, level of class and college in which a course was offered had on feature adoption (only class size had significant impact). Hornik et al (2008) have put courses into high and low paradigms and seen how this, plus the level of the course, has impacted on outcomes in web-based courses. There are some limitations of this work we might be able to fill. For example, Malikowski et al (2006) manually checked courses sites and because of this are limited to observations from a single term.

Second, we’ve already done some work in this area in our first paper. We

• Looked at how two different LMS within the same institution at the same time could have very different levels of feature adoption over time.
• Discovered an apparent anomaly in the previously identified correlation between activity within an LMS and student grades.
• Showed how a course designed with an instructional designer has a fairly different signature than other courses.
• Showed how increased teaching staff participation in an LMS tends to increase the connection between student LMS activity and final grades.

This sort of examination of external factors and their impact on LMS usage is useful as it helps identify areas of interest in terms of further research and also potential insights for course design. It’s also (IMHO) somewhat useful in its own right without any need for additional research. So it’s something relatively easy for us to do, but also should be fairly easy for others to experiment with.

Abstracting this work up a bit

The first step in examining this might be an attempt to abstract out the basic principles and components of this sort of work. If we can establish some sort of pattern/abstraction this can guide us in the type of work required and some sort of move towards a more rigorous process. The following is my initial attempt.

There have been two main approaches we’ve taken in the first paper:

1. Impacts on student performance.
2. Impacts on LMS feature adoption.

Impacts on student performance

An example is the impact of an instructional designer. The following graph compares the level of student participation mapped against final result between course designed with an instructional designer and all other courses.

Instuctional Designer Designed Courses vs Overall Average

In this type of example, we’ve tended to use three main components:

1. A measure of LMS usage.
   So far we have concentrated on

   the average number of hits by the student on the course website and discussion forum; and

   • the average number of posts and replies by the student on the discussion forum
2. A measure of student performance.
   Limited to grade achieved in the course, at the moment.
3. A way to group students.
   This has been done on the basis of mode of delivery/type of student (i.e. a distance education student, an Australian on-campus student, an international student) or by different types of courses.

Having identified these three components we can actively search for alternatives. What alternatives to student performance might there be?

For example, in the paper we use Fresen’s (2007) taxonomy of factors to promote quality web-supported learning as a way to group students. For example, staff participation should promote quality, hence is there any difference in courses with differing levels of staff participation?

Are there other theoretical insights which could guide this work?

Impacts on LMS feature adoption

We’ve used the LMS independent framework for LMS features developed by Malikowski et al (2007) to examine to what level different features are used within courses. We’ve looked at this over time and between different LMS. The following shows the evolution of feature adoption over time within the Blackboard LMS used at CQU.

Blackboard Feature Adoption

Under this model, the components could be described as:

• Framework for grouping LMS features.
• Definition of adoption.

A mixture of the two?

I wonder if there’s any value in using the level of feature adoption as another way of grouping courses to identify if there’s any connection with student outcome. e.g. do courses with just content distribution have different student outcomes/usage than courses with everything?

Next steps

Some quick ideas:

• Look at improving the abstraction and alternatives of the two abstractions above.
• Look at focusing on developing some platform independent database schema to enable the cross-LMS and cross-institutional comparison of the above two abstractions.
  This would include:
  • the database scheme;
  • some scripts to convert various LMS logs into that database format;

  some tools to automate interseting graphs.

References

Fresen, J. (2007). “A taxonomy of factors to promote quality web-supported learning.” International Journal on E-Learning 6(3): 351-362.

Hornik, S., C. S. Saunders, et al. (2008). “The impact of paradigm development and course level on performance in technology-mediated learning environments.” Informing Science 11: 35-58.

Malikowski, S., M. Thompson, et al. (2006). “External factors associated with adopting a CMS in resident college courses.” Internet and Higher Education 9(3): 163-174.

Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

After at least a decade of “wouldn’t it be a good idea if” and at least one aborted attempt (hint: an organisational restructure in which you are a loser, is not a great context for a new project with intra-organisational implications), the Indicators Project is getting started. This post is my attempt to define what the project means for me. What I hope to get out of the project, and what I hope others might get out of the project.

The main aim is to let people know about the project and encourage feedback, either here or on the project blog.

Absence of data and poor decision making

Dave Snowden defines sensemaking as

How do we make sense of the world, so that we can act in it.

When it comes to education, David Wiley makes this point which resonates strongly with me

The data that we, educators, gather and utilize is all but garbage. What passes for data for practicing educators? An aggregate score in a column in a gradebook. A massive, course-grained rolling up of dozens or hundreds of items into a single, collapsed, almost meaningless score. “Test 2: 87.” What teacher maintains item-level data for the exams they give? What teacher keeps this data semester to semester, year-to year? What teacher ever goes back and reviews this historical data?

For a long time I have believed that the absence and/or poor quality of the data available has meant that universities have been particularly bad at sensemaking around learning and teaching, and especially e-learning.

For me, a major consequence of this “garbage data” is that decisions made within universities (I work within a university, I’m paid to help improve learning and teaching at that institution, so my focus is on universities) about learning and teaching, and especially about e-learning, are made with very little sense of what is happening within the real world. This situation is increasingly getting worse as, at least within my experience, management at universities are attempting to adopt a more top-down, “corporate” approach to decision making.

Such an approach to decision making means that when management make the decisions about learning and teaching not only don’t have good data to base their decision on. They are also making decisions on the basis of one of the following categories of teaching experience:

• only taught recently with a significant amount of support;
  (this means they don’t have to experience all the low level “make work” that consumes so much time)
• haven’t taught for a number of years;
• have never taught within the local context; or
• have never taught.

For individual academics, they are stuck with the “garbage data” from their own courses and their own gut feel. Since teaching at University is mostly a solo activity, there is little or no opportunity to compare and contrast with the experience of others. Even when the opportunity does arise, it has to be done with “garbage data”.

Support staff, be they instructional designers, academic developers or IT folk, are almost entirely without data, which means they can’t target their assistance. They have to take a one size fits all (i.e. one size that helps no-one) approach. Mainly because what data that is available about learning and teaching is only available to the teacher or their line supervisor.

Students, well they are at the bottom of the pile. They get essentially no indication of how where the sit with respect to other students. etc.

The indicators project aims to provide better data to teaching staff, management, support staff and students.

What can be done?

David Wiley believes that

using technology to capture, manage, and visualize educational data in support of teacher decision making has the potential to vastly improve the effectiveness of education.

A lot of the work by Dave Snowden is based around the idea of achieving

A synthesis of technology and human intelligence

Using technology for what it is good for in order to generate indicators that can help people do what they are good at – pattern matching.

David Wiley’s long term goal is huge, difficult and expensive. You can read more about it on his blog. That goal is beyond the scope of our little indicators project. I think the aims for our project can be summarised as:

• Identify potentially interesting indicators from LMS usage data and some other institutional data (e.g. student characteristics etc.).
• Make that information available to students, teaching staff, management, support staff and researchers.
  We aren’t likely to achieve all this at once, different folk will get it at different times.
• Engage in additional research around the indicators, how they are used, how they can be used and what they can tell us about learning and teaching.
• Return to step #1.

Cross platform and cross institution

Importantly, we’re aiming to/hoping for the project to identify, encourage and enable use of the indicators across different institutions and different LMS. As we progress, we’ll be looking for people interest in partnering with the project.

Graphical representation

In an attempt to understand what we have to do and where the interesting work might be we developed the following graphical representation of the project.

Figure 1. Project Overview

Working from the bottom up, the figure includes:

• LMS and institutional specific data.
  Each institution will have its own LMS and also some other data in the form of information about the students (e.g. age, country of origin, type of student) and the courses (e.g. discipline, number of campuses offered at etc.).
• We need to do some “research” to identify the knowledge necessary to effectively convert this institutional and LMS dependent data into something that is independent of LMS and institution.
• The LMS & institutional independent data forms the main data source of the indicators. At the very least, partner institutions will be able to perform comparisons. In a perfect world, the data will be in such a form as to enable free sharing, anyone who has an interest can get the data and perform analysis.
• We then need to do some research to generate knowledge to convert the LMS and institution independent data into indicators. The indicators abstract the data into a form that provides useful knowledge for students, teachers, managers, support staff or researchers.
  One simple example, is the percentage of courses within an LMS that have adopted specific features.
• Some of the “useful knowledge” will be passed onto the institutional business intelligence folk who are responsible for institutional data warehouses, dashboards and the like.
• Some of the useful knowledge will be used by a variety of people (teaching staff, support staff, students and management) to improve the practice of learning and teaching.
• Some of the useful knowledge will be used as the basis for additional research to identify the whys and wherefores of the indicators.
  For example, Why do international students “break” the link between LMS activity and student grades?

Problems

This is by no means a simple task. There are any number of problems that will impact the project. Here are some.

Online only is rare

David Wiley is in the somewhat rare situation of having an online only context

The Open High School of Utah is the first context in which I’m studying this use of technology. Because it is an online high school, every interaction students have with content (the order in which they view resources, the time they spend viewing them, the things they skip, etc.) and every interaction they have with assessments (the time they spend answering them, their success in answering them, etc.) can all be captured and leveraged to support teachers.

It is very rare for my institution, and I’m assuming many other universities, to have courses that are entirely online. In our situation a large percentage of our students must attend on-campus sessions and another large percentage believes they are missing out on something important if they don’t get face-to-face. So, in our situation the online data is only ever going to tell part of the story. It is going to have to be supplemented with other approaches and methods.

Data quality

David Warlick in a blog post that responds to David Wiley’s post (is it me, or are there a lot of David’s in this post?) identifies the problems with data quality

even in the best of situations, the data is scarce, shallow, grainy, and awfully expensive to collect

He is perhaps talking about a different context with high schools, but some of these limitations apply in existing work. Much of the research into LMS usage has focused on the use of surveys, interviews of manual examination of course sites to generate insight. Where data mining is done on system data it is often for limited time frames (e.g. 1 term or 1 year) and is usually communicated in a LMS dependent way that makes comparisons between systems and institutions difficult.

Who will use it?

David Warlick makes another important point in his blog post. This time the question is “who will use all this data?”

not to mention that the only people who can make much use of it are the data dudes that school systems have been hiring over the past few years.

This is a problem I’ve seen with universities with the rise of data warehouses and dashboard. Unless there is a particular motivated and well resourced team, such information systems become the toys of the “data dudes”, occasionally the weapons of managers who wish to make a particular point, or a resource for a small group of researchers to publish papers. They rarely become embedded into the day to day practice of learning and teaching.

The LMS problem

The LMS is based on the assumption that all “learning” – or at least content access and discussion forum use – occurs within the LMS. This “one ring to rule them all” approach does provide one benefit. All of this data is in the one place, the one system, the one database.

This “one ring to rule them all” approach is also, in my opinion and that of many others, the main problem with the LMS. It removes choice from the student and the teacher about what tools can be used. However, if alternatives such as personal learning environments become prevalent, then the sort of approach being adopted by the indicators project will no longer be possible. The focus will have to change to the type of question Stephen Downes raised when pointing to Wiley’s post

Shouldn’t we be devising ways for students to organize and track their own learning?

This an important point. If I had my way we wouldn’t be using an LMS. The trouble is that the LMS is the almost universal response to e-learning by universities. To get them to change, we’re going to have to – at the very least – provide lots of meaningful data that encourage management and others to recognise the limitations of the LMS approach. Certainly one of my aims in being involved with the indicators project is to illustrate the inherent limitations and problems with the LMS approach.

Where to from here?

The project is starting to gather some momentum. We’ve had our first paper accepted at a conference. We’re talking about research and ALTC grants. We’ve started identifying additional work we need to make progress on, in particular making a start on the cross LMS comparisons. We’re talking about making connections with various folk to help the project move one.

So, feel free to share your comments and thoughts.

I recently posted about the need to develop an approach that allows for the simple and consistent comparison of usage and feature adoption between different Learning Management Systems (aka LMS, Virtual Learning Environments – VLEs – see What is an LMS?). That last post on the need didn’t really establish the need. The aim of this post is to explain the need and make some first steps in identifying how you might go about enabling this sort of comparison.

The main aim is to get my colleagues in this project thinking and writing about what they think we should and how we might do it.

What are you talking about?

Just to be clear, what I’m trying to get at is a simple method by which University X can compare how its staff and students are using its LMS with usage at University Y. The LMS at University Y might be different to that at University X. It might be the same.

They might find out that more students use discussion forums at University X. More courses at University Y might use quizzes. The could compare the number of times students visit course sites, or whether there is a correlation between contributions to a discussion forum and final grade.

Why?

The main reason is so that the university, its management, staff, students and stakeholders have some idea about how the system is being used. Especially in comparison with other universities or LMSes. This information could be used to guide decision making, identify areas for further investigation, as input into professional development programs or curriculum design projects, comparison and selection processes for a new LMS, and many other decisions.

There is a research project coming out of Portugal that has some additional questions that are somewhat related.

The main reason is that there currently appears to be no simple, effective method for comparing LMS usage between systems and institutions. The different assumptions, terms and models used by systems and institutions get in the way of appropriate comparisons.

How might it work?

At the moment, I am thinking that you need the following:

• a model;
  An cross-platform representation of the data required to do the comparison. In the last post the model by Malikowski et al (2007) was mentioned. It’s a good start, but has doesn’t cover everything.

  As a first crack the model might include the following sets of information:

  • LMS usage data;
    Information about the visits, downloads, posts, replies, quiz attempts etc. This would have to be identified by tool because what you do with a file is different from a discussion forum, from a quiz etc.
  • course site data;
    For each course, how many files, is there a discussion forum, what discipline is the course, who are the staff, how many students etc.
  • student characteristics data;
    How were they studying, distance education, on-campus. How old were they?
• a format;
  The model has to be in an electronic format that can be manipulated by software. The format would have to enable all the comparisons and analysis desired but maintain anonymity of the individuals and the courses.
• conversion scripts; and
  i.e. an automated way to take institutional and LMS data stick it into the format. Conversion scripts are likely to be based around LMS and perhaps student records system. e.g. a Moodle conversion script could be used by all the institutions using Moodle.
• comparison/analysis scripts/code.
  Whatever code/systems are required to take the information in the format and generate reports etc. that help inform decision making.

Format

I can hear some IT folk crying out for a data warehouse to be used as the format. The trouble is that there are different data warehouses and not all institution’s would have them. I believe you’d want to initially aim for a lowest common denominator, have the data in that and then allow further customisation if desired.

When it comes to the storage, manipulation and retrieval of this sort of data, I’m assuming that a relational database is the most appropriate lowest common denominator. This suggests that the initial “format” would be an SQL schema.

How would you do it?

There are two basic approaches to developing something like this:

• big up front design; or
  Spend years analysing everything you might want to include, spend more time designing the perfect system and finally get it ready for use. Commonly used in most information technology projects and I personally think it’s only appropriate for a very small subset of projects.
• agile/emergent development.
  Identify the smallest bit of meaningful work you can do. Do that in a way that is flexible and easy to change. Get people using it. Learn from both doing it and using it to inform the next iteration.

In our case, we’ve already done some work from two different systems for two different needs. I think discussion forums are shaping up as the next space we both need to look at, again for different reasons. So, my suggestion would be focus on discussion forums and try the following process:

• literature review;
  Gather the literature and systems that have been written analysing discussion forums. Both L&T and external. Establish what data they require to perform their analysis.
• systems analysis;
  Look at the various discussion forum systems we have access to and identify what data they store.
• synthesize;
  Combine all the requirements from the first two steps into some meaningful collection.
• peer review;
  If possible get people who know something to look at it.
• design a database;
  Take the “model” and turn it into a “format”.
• populate the database;
  Write some conversion scripts that will take data form the existing LMSes we’re examining and populate the database.
• do some analysis;
  Draw on the literature review to identify the types of analysis/comparison that would be meaningful. Write scripts to perform that role.
• reflect on what worked and repeat;
  Tweak the above on the basis of what we’ve learned.
• publish;
  Get what we’ve done out in the literature/blogosphere for further comment and criticism.
• attempt to gather partners.
  While we can compare two or three different LMS within the one institution. The next obvious step would be to work with some other institutions and see what insights they can share.

The knowledge and experience gained this for “discussion forums” could then be used to move onto other aspects.

What next?

We probably need to look at the following:

• See if we can generate some outside interest.
• Tweak the above ideas to get something usable.
• Gather and share a bibliography of papers/work around analysing discussion forum participation.
• Examine the discussion forum data/schema for Blackboard 6.3 and Webfuse.

That’s probably enough to be getting on about.

References

Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

Some colleagues and I are putting the finishing touches on a paper that has arisen out of the indicators project. The paper is an exploratory paper, seeking to find interesting patterns that might indicate good or bad things about the use of LMS (learning management systems, aka course management systems, virtual learning environments etc) that might help improve decision-making by all participants (students through management). I hope to post the paper in coming days.

This post is about one aspect of the paper. The section where we compare feature adoption between two different LMS that have been used side-by-side at our institution: Blackboard and Webfuse. (Important: I don’t believe Webfuse is an LMS and will argue that in my PhD (Webfuse is the topic of my thesis). But it’s easier to go with the flow). This is one of the apparent holes in the literature, we haven’t found any publications analysing and comparing system logs from different LMS, especially within the one institution over the same long time frame. In our case we went from 2005 through the first half of 2009.

The aim of this post is to identify the need and argue for the benefits in developing a LMS independent means of analysing and comparing the usage logs of different LMS at different institutions.

Anyone interested?

The following gives a bit of the background, reports on some initial findings and out of that identifies the need for additional work.

Our first step

Blackboard and Webfuse have a number of significant differences. All LMS have somewhat different assumptions, designs and names. Webfuse is significantly different, but that’s another story. The differences make comparisons between LMS more difficult. How do you compare apples with apples?

The only published approach we’re aware of that attempts to make a first step towards a solution to this problem is the paper by Malikowski, Thompson and Theis (2007) for which the abstract makes the following claims

…This article recommends a model for CMS research that equally considers technical features and research about how people learn…..This model should also ease the process of synthesizing research in CMSs created by different vendors, which contain similar features but label them differently.

I’ve talked about and used the model previously (first, second and other places). For the purposes of the paper we produced a different representation of the Malikowski et al (2007) model.

From my perspective there are three contributions the model makes

1. Provides an argument for 5 categories of features and LMS might have, gives them a common title and specifies which common features fit where.
2. Draws on existing literature give some initial benchmarks for the level of adoption (specified by the percentage of courses with a feature) to be expected grouped into three levels.
   I must admit that Malikowski et al don’t specify the percentages directly, these are taken from the examples they list in tables.
3. Suggests a model where features are adopted sequentially over time as academics become more comfortable with existing features.

Blackboard versus Webfuse – 2005 to 2009

The benefit the model has provided us is the ability to group the different features of Webfuse and Blackboard into the five categories and then compare the levels of feature adoption between the two systems and with the benchmarks identified in the Malikowksi et al (2007) paper. The following summarises what we found.

Transmitting content

Malikowski et al (2007) define this to include announcements, uploaded files and the use of the gradebook to share grades (but not assignment submission etc.). The following graph shows the percentage of course sites in both Blackboard (black continuous line), Webfuse (black dashed lines) and the “benchmark region” identified in Malikowski et al. In the case of transmitting content the “benchmark region” is between 50 and 100%.

This shows that both Blackboard and Webfuse are in the “benchmark region”. Not surprising given the pre-dominant use of LMSs for content transmission. What may be surprising is that Webfuse only averages around 60-75%. This is due to one of those differences in LMS. Webfuse is designed to automatically create default course sites that contain a range of content. Also, it’s quite common for the announcements facility in a Webfuse course site to be used by faculty management to disseminate administrative announcements to students.

So, in reality 100% of Webfuse courses transmit content. The percentage show those courses where the academics have uploaded additional content or made announcements themselves.

Class interactions

Class interactions covers chat rooms, email, discussion forums, mailing lists etc. Anything that get folk in a course talking.

Both Blackboard and Webfuse are, to varying extents, outside of the “benchmark area”. Webfuse quite considerably reaching levels near 100% in recent years. Blackboard has only just crept over. This creeping over of Bb may be an indicator that the “benchmark area” is out of date. It was created drawing on 2004 and earlier literature. If feature adoption increases over time, the “benchmark area” has probably moved up.

Evaluating students

Online assignment submission, quizzes and use of other tools to assess/evaluate students.

Over recent years Webfuse has seen double the adoption of these features than Blackboard. It’s grown outside the “benchmark area”. Most of this is online assignment submission, in fact some of the courses using the Webfuse online assignment submission system are actually Blackboard courses.

Evaluating course/instructor

The last category we covered was evaluating the course/instruction through survey tools etc. We didn’t cover computer-based instruction as very few Blackboard courses use it and Webfuse doesn’t provide the facility.

Which raises an interesting question. I clearly remember a non-Webfuse person being quite critical that Webfuse did not offer the computer-based instruction funtionality – we could have added it but no-one ever asked. What is better, paying for features few people ever use or not having features that a few people will use?

First, it should be pointed out that for “rarely used” features like course evaluation there is an absence of percentages in Malikowski et al (2007). I’ve specified 20% as the upper limit for this “benchmark area” because “moderately used” was 20% or higher. So it’s probably unfair to describe the Blackboard adoption level as being at the bottom of the range. On the hand, Webfuse is streets ahead. Near 100% dropping to just less than 40%. More on this below.

Work to do

Generating the above has identified a need or value in the following future work:

• Do an up to date literature review and establish a new “benchmark area”.
  Malikowski et al (2007) rely on literature from 2004 and before. Levels of adoption have probably gone up since then.
• Refine the list of features per category through the same literature review.
  In recent years LMS have added blogs, wikis, social networking etc. Where do they fit?
• Refine the definition of “adoption”.
  Malikowski and his co-authors have used at least two very different definitions of adoption. There is apparently no work to check that the papers used to illustrate the model in Malikoswki et al (2007) use a common definition of adoption.
• Develop feature specific LMS independent usage descriptions.
  In their first paper Malikowski et al (2006) count adoption as the presence of a feature, regardless of how broadly it is used. This causes problems, for example, the course evaluation figure for Webfuse is near 100% because for a number of years a course barometer (Jones, 2002) was a standard part of a Webfuse default site. i.e. everyone course had one. Just doing a quick check, only 23% of Webfuse courses in 2006 had a barometer in which a student made a comment.

  Malikowski (2008) adopted a new measure for adoption. Course use of a particular feature had to be above the 25th percentile of use for that feature in order to be counted. I don’t find this a good measure. Just 1 student comment on a barometer could be a potentially essential use of the feature.

  There appears to be a need for being able to judge the level of use of a feature in a way that is sensitive to the feature. 1 entry in a gradebook for a course for 500 students is probably an error can be ignored. 1 comment on a barometer for that same course that points out an important issue probably shouldn’t be ignored.

• Attempt to automate comparison between LMS.
  In order to enable broader and quicker comparison between different LMS, whether between institutions or within institutions, there appears a need to automate the process. To make it quicker and more staight forward.

  One approach might be to design a LMS independent database scheme for the extended Malikowski et al (2007) model. Such a scheme would enable people to write “conversion scripts” that take usage logs from Blackboard, Moodle, WebCT or whatever LMS and automatically insert them into the schema. Once someone has written the conversion script of an LMS, no-one else would have to. The LMS independent schema could than be analysed and used to compare and contrast different systems and different institutions without the apples and oranges problem.

References

Jones, D. (2002). Student Feedback, Anonymity, Observable Change and Course Barometers. World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado, AACE.

Malikowski, S., M. Thompson, et al. (2006). “External factors associated with adopting a CMS in resident college courses.” Internet and Higher Education 9(3): 163-174.

Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

As part of the thesis I’ve been engaging with some of the literature around LMS feature usage to evaluate usage of Webfuse. A good first stab of this was reported in an earlier post. There were a number of limitations of that work, it’s time to expand a bit on it. To some extent for the PhD and to some extent because of a paper.

As with some of the other posts this one is essentially a journal or a log of what I’m doing and why I’m doing it. A permanent record of my thinking so I can come back later, if needed.

There’s even an unexpected connection with power law distributions towards the end.

Content distribution

In that previous post I did not include a graph/any figures around the use of Webfuse course sites to distribute content or files. This is because Webfuse had a concept of a default course site. i.e. every course would have the same basic default site created automatically. Since about 2001 this meant that every course site performed some aspect of information distribution including: the course synopsis on the home page, details about the course assessment, details about course resources including textbook details and a link to the course profile, and details about the teaching staff.

Beyond this staff were able to upload files and other content as they desired. i.e. moving beyond the default course site was optional and left entirely up to the teaching staff. Some of us, perhaps went overboard. Other staff may have been more minimal. The aim here is to develop metrics that illustrate that variability.

Malikowski et al (2007) have a category of LMS usage called Transmitting Content. The LMS features they include in this category include:

• Files uploaded into the LMS.
• Announcements posted to the course site.

So, in keeping with the idea of building on existing literature. I’ll aim to generate data around those figures. Translating those into Webfuse should be fairly straight forward, thinking includes:

• Files uploaded into the LMS.
  Malikowski et al (2007) include both HTML files and other file types. For Webfuse and its default course sites I believe I’ll need to treat these a little differently:
  • HTML files.
    The default course sites produce HTML. I’ll need to exclude these standard HTML files.
  • Other files.
    Should be able to simply count them.
  • Real course sites.
    Webfuse also had the idea of a real course site. i.e. an empty directory into which the course coordinator could upload their own course website. This was usually used by academics teaching multimedia, but also some others, who knew what they wanted to do and didn’t like the limitations of Webfuse.
• Announcements.
  The default course site has an RSS based announcements facility. However, some of the announcements are made be “management”. i.e. not the academics teaching the course but the middle managers responsible for a group of courses. These announcements are more administrative and apply to all students (so they get repeated in every course). In some courses they may be the only updates. These announcements are usually posted by the “webmaster”, so I’ll need to exclude those.

Implementation

I’ll treat each of these as somewhat separate.

• Calculate # non-HTML files.
• Calculate # of announcements – both webmaster and not.
• Calculate # HTML files beyond default course site (I’ll postpone doing this one until later)

Calculate # non-HTML files.

Webfuse created/managed websites. So all of the files uploaded by staff exist within a traditional file system. Not in a database. With a bit of UNIX command line magic it’s easy to exact name of every file within a course site and remove those that aren’t of interest. The resulting list of files is the main data source that can then be manipulated.

The command to generate the main data source goes like this

find T1 T2 T3 -type f | get all the files for the given terms
grep -v ‘.htm$’ | grep -v ‘.html$’ | remove the HTML files
grep -v ‘CONTENT$’ | remove the Webfuse data files
grep -v .htaccess | remove Apache access restriction file
grep -v ‘updates.rss$’ | remove the RSS file used for announcements
grep -v ‘.ctb$’| grep -v ‘.ttl$’ | grep -v ‘/Boards/[^/]*$’ | grep -v ‘/Members/[^/]*$’ | grep -v ‘/Messages/[^/]*$’ | grep -v ‘/Variables/[^/]*$’ | grep -v ‘Settings.pl’ | remove files created by discussion forum
sed -e ‘1,$s/.gz$//’

The sed command at the end removes the gzip extension that has been placed on all the files in old course sites that have been archived – compressed.

The output of this command is the following

T1/COIT11133/Assessment/Assignment_2/small2.exe
T1/COIT11133/Assessment/Weekly_Tests/Results/QuizResults.xls
T1/COIT11133/Resources/ass2.exe

The next aim is to generate a file that contains the number of files for each course offering. From there the number of courses with 0 files can be identified, as can some other information. The command to do this is

sed -e ‘1,$s/^\(T.\/………\/\).*$/\1/’ all.Course.Files | sort | uniq -c | sort -r -n > count.Course.Files

After deleting a few entries for backup or temp directories. We have our list. Time to manipulate the data, turn it into a CSV file and into Excel. Graph below, fairly significant disparity in number of files – the type of curve looks very familiar though.

In total, for 2005 there were 178 course sites that had files. That’s out of 299 – so 59.5%. This compares to the 50% that Col found for the Blackboard course sites in the same year.

Calculate # of Announcements

The UNIX command line alone will not solve this problem. Actually, think again, it might. What I have to do is:

• For each updates.rss
  • count the number of posts by webmaster
  • count the number of posts by non-webmaster
  • output – courseOffering,#webmaster,#non-webmaster

Yep, a simple shell script will do it

echo COURSE,ALL,webmaster
for name in `find T1 -name updates.rss`
do
  all=`grep '' $name | wc -l`
  webmaster=`grep 'webmaster' $name | wc -l`
  echo "$name,$all,$webmaster"
done

Let’s have a look at the 2005 data. Remove some dummy data, remove extra whitespace. 100% of the courses had updates. 166 (55%) had no updates from the teaching staff, 133 (45%) did. That compares to 77% in Blackboard. Wonder if the Blackboard updates also included “webmaster” type updates?

In terms of the number of announcements contributed by the teaching staff. The following graph shows the distribution. The largest number for a single offering was 34. Based on a 12 week CQU teaching term, that’s almost, on average, 3 announcements a week

Power laws and LMS usage?

The two graphs above look very much like a power law distribution. Clay Shirky has been writing and talking about power law distributions for some time. Given that there appears to be a power law distribution going on here with usage of these two LMS features, and potentially that the same power law distribution might exist with other LMS features, what can Shirky and other theoretical writings around power law distributions tell us about LMS usage?

References

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Over the last week I’ve posted a bit around papers and ideas about how you evaluate the use of a learning management system (LMS). This post is intended to summarise findings from an initial evaluation of the use of features in the Webfuse system from 1997 through 2009. Webfuse is the system I designed as part of my PhD and has been used at Central Queensland University since 1997. At least the “LMS” part of Webfuse will cease being used in 2010.

Aside: Webfuse is not an LMS. Anyone who calls Webfuse an LMS doesn’t get it.

Overall, I believe these figures, even with their limitations, show that the processes and ideas underpinning Webfuse have contributed to a very different result than is expected from the literature. I would argue that the Webfuse approach, in a context of some significant difficulty, has produced a result with some significant benefits.

About the figures

The figures presented below results from an initial analysis of the archives. The figures presented below concentrate on the number of course sites supported by Webfuse and the percentage of those courses that contained various Webfuse features. The features are grouped into the categories proposed by Malikowski, Thompson and Theis (2007) and illustrated in the figure below adapted from Malikowski et al (2007).

An important point made in the following is related to usage levels embedded in the Malkowski et al (2007) model. The model groups LMS functionality into five categories and then, drawing on the results reported in the literature, break the five categories into three levels of usage:

• Level 1 – Most used category.
  Only the transmitting content category is put into this level. This reflects findings that by far the overwhelming use of an LMS has been to distribute information.
• Level 2 – Moderately used category.
  Student evaluation (online quizzes, student assessment) and class interactions fit into this category.
• Level 3 – Rarely used category.
  Evaluating the course/instructors and computer-based instruction fit here.

Malikowski et al in their writings have used two different approaches to defining adoption. The first defines adoption as the presence of the functionality and doesn’t take into account how much the feature is actually used. The second required the feature usage to be greater than the 25 percentile. The following uses the first, it’s easier to calculate and doesn’t have some of the problems of the latter

I’ve included some brief comments with each graph.

Webfuse course sites

• I believe that the drop in course sites in 2000 was due to some coordinators adopting WebCT which had been introduced as the “institutional” system. Within an offering or two, these courses had returned to Webfuse.
• 1997 through 2000 were the naive years of Webfuse. The level of features in Webfuse and the default course sites was low. Familiarity with online learning with staff was relatively low.
• At the end of 2000 the golden years of the Webfuse team commenced. I worked full-time on Webfuse, there was a growing and talented Webfuse team, the underlying technology was getting better and there was growing need.
• The middle of 2001 saw the introduction of a new “default course site” that offered more scaffolding for staff than earlier versions. It’s essentially the version still being used today.
• After 2001 there was never a great deal of work done on expanding the functionality of the course sites as a whole, beyond the odd bit of tinkering and fixing.
• By 2004 the faculty decided there was no value in me working on Webfuse. Around this time Blackboard was introduced and not long afterwards there was faculty restructure.
• By 2006 the Webfuse team moved out of a faculty into the central IT division. Not long after the team was down to 1 and a bit people.

Transmitting content

I have not bothered to include a graph showing the percentage of Webfuse courses transmitting content. The Webfuse “model” was that a default course site was automatically created for all courses. By default, this included a link to the course profile, some information about the course and the teaching staff.

So, from the start 100% of the Webfuse course sites were used to transmit content.

Class interactions

• In 1997 Webfuse was used only for courses within the Department of Mathematics and Computing. There was a history of using mailing lists in some courses. I’m not sure all of these are included in the figure from 1997.
• From 1998, Webfuse supported courses within the Faculty of Informatics and Communication and included courses in Journalism, Communications, Multimedia and others.
• I was surprised when I found that the level of class interactions rose to 100% in the last few years. This is a significantly different result than that reported in the literature and breaks Malikoswki et al’s (2007) suggested model.

Evaluating students

• Getting into the 60/70% of courses using Webfuse to evaluate students is another result somewhat different than you would expect, certainly when compared to the level of adoption in other institutional systems at CQU.
• A significant proportion of this is online assignment submission which has been a focus of work within Webfuse and has resulted in publications (Jones and Jamieson, 1997; Jones and Behrens, 2003; Jones et al, 2005) and a system that has been streets ahead of what’s been available in LMSes.

Evaluating courses or staff

• Since 1997 Webfuse has provided a feature to survey students. Initially this was based on the paper-based course evaluation form used at CQU at that time. Preceding the institutional use of online forms to survey students by a number of years.
• The upsurge in course evaluation starting in 1999 arises because of inclusion of the course barometer feature. In 1999, I attended the EdMedia’99 conference in Seattle. One of the presentations I attended introduced the idea of the course barometer (Svensson, Andersson, Gadd and Johnsson, 1999). I thought it would be great for my course and created a Webfuse course barometer. In that first year a handful of other academic voluntarily added it to their sites.
• In June 2001, the faculty QA person convinced me and faculty management that it would be a good day to make the barometer a part of the default course site. i.e. every course site would have one. This continued until 2006 when it was made optional.
• Only for a brief period of time (first six months of 2002) was the barometer really effectively supported by the faculty. However, due to the nature of the barometer and some other contextual issues, it wasn’t an entirely positive experience. For more information refer to a couple of papers that were written (Jones, 2002; 2003)
• It is likely when analysis of usage moves beyond the presence of a barometer in a course site, that percentages might decrease. However, the level of usage in the last few years, when it has been optional, is still significantly higher than might be expected based on the literature.

Computer-based instruction

Webfuse has never provided any functionality that fits within this category.

References

Jones, D., & Jamieson, B. (1997). Three Generations of Online Assignment Management. Paper presented at the ASCILITE’97, Perth, Australia.

Jones, D. (2002). Student Feedback, Anonymity, Observable Change and Course Barometers. Paper presented at the World Confernece on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado.

Jones, D. (2003). Course Barometers: Lessons gained from the widespread use of anonymous online formative evaluation. Paper presented at the OLT’2003, QUT, Brisbane.

Jones, D., & Behrens, S. (2003). Online Assignment Management: An Evolutionary Tale. Paper presented at the 36th Annual Hawaii International Conference on System Sciences, Hawaii.

Jones, D., Cranston, M., Behrens, S., & Jamieson, K. (2005). What makes ICT implementation successful: A case study of online assignment submission. Paper presented at the ODLAA’2005, Adelaide.

Malikowski, S., Thompson, M., & Theis, J. (2007). A model for research into course management systems: bridging technology and learning theory. Journal of Educational Computing Research, 36(2), 149-173.

Svensson, L., Andersson, R., Gadd, M., & Johnsson, A. (1999). Course-Barometer: Compensating for the loss of informal feedback in distance education. Paper presented at the EdMedia’99, Seattle, Washington.