MAV, #moodle, process analytics and how I’m an idiot

I’m currently analysing the structure of a course I teach and have been using @damoclarky’s Moodle Activity Viewer to help with that. In the process, I’ve discovered that I’m an idiot in having missed the much more interesting and useful application of MAV than what I’ve mentioned previously. The following explains (at least one example of) how I’m an idiot and how MAV can help provide a type of process analytics as defined by Lockyer et al (2013).

Process analytics

In summary, Lockyer et al (2013) define process analytics as one of two broad categories of learning analtyics that can help inform learning design. Process analytics provide insight into “learner information processing and knowledge application … within the tasks that the student completes as part of a learning design” (Lockyer et al, 2013, p. 1448). As an example, they mention the use of social network analysis of student discussion activity to gain insights into engaged a student is with the activity and who the student is connecting with within the forum.

The idea is that a learning analytics application becomes really useful when combined with the pedagogical intent of the person who designed the activity. The numbers and pretty pictures by themselves are more valuable in combination with teacher knowledge.

A MAV example – Introduction discussion forum

I’m currently looking through the last offering of my course, trying to figure out what worked and what needs to be changed. As part of this, I’m drawing on MAV to give me some idea of how many students clicked on particular parts of the course site and how many times they did click. At this level, MAV is an example of a very primitive type of learning analytics.

Up until now, I’ve been using MAV to look at the course home page as captured in this large screen shot. When I talk about MAV, this is what I show people. But now that I actually have MAV on a computer where I can play with it, I’ve discovered that MAV actually generates an access heat map on any page produced by Moodle.

This includes discussion forums, as shown in the following image (click on it to see a larger version).

Forum students by David T Jones, on Flickr

This is a modified (I’ve blurred out the names of students) capture of the Introduction discussion forum from week 1 of the course. This is where students are meant to post a brief introduction to themselves, including a link to their newly minted blog.

With a standard Moodle discussion forum, you can see information such as: how many replies to each thread; who started the thread; and, who made the last post. What Moodle doesn’t show you is how many students have viewed those introductions. Given the pedagogical purpose of this activity is for students to read about other students, knowing if they are actually even looking at the posts is useful information.

MAV provides that information. The above image is MAV’s representation of the forum showing the number of students who have clicked each link. The following image is MAV’s representation of the number of clicks on each link.

Forum clicks by David T Jones, on Flickr

What can I derive from these images by combining the “analytics” of MAV with my knowledge of the pedagogical intent?

  • Late posts really didn’t help make connections.

    The forum is showing the posts from most recent to least recent. i.e. the posts near the top are the late posts. This forum is part of week 1, which was 15th to 19th of July, 2013. The most recent reply (someone posting their introduction) was made in Oct. Subsequent posts are from 7th to 10th August, almost a month after the task was initially due (the first assignment was due 12th August, completing this task contributed a small part of the mark for the first assignment).

    These late posts had really very limited views. No more than 4 students viewing them.

  • But then neither did many of them.

    Beyond the main thread started by my introduction, the most “popular” other introduction was clicked on 41 times by 22 students (out of 91 in the course). Most were significantly less than this.

    Students appear not to place any importance on reading the introductions of others. i.e. the intent is not being achieved.

  • Students didn’t bother looking at my Moodle profile.

    The right hand column of the images shows the name of the author and the time/date of the last post in a thread. The author’s name is also a link to their Moodle profile.

    MAV has generated an access heat map for all the links, including these. There are no clicks on my profile link. This may be because the course site has a specific “Meet the teaching team” page, or it maybe they simply don’t care about learning more about me.

  • It appears students who posted in a timely manner had more people looking at their profiles.

    This is a bit of stretch, but the folk who provided the last post to messages toward the bottom of the above images tend to have higher clicks on their profile than those later in the semester. For example, 19, 22, and 12 for the three students providing the last posts for the earliest posts, and, 1, 1, and 7 for the students providing the last post for the more recent posts.

  • Should I limit this forum to one thread?

    The most popular thread is the one containing my introduction (549 clicks, 87 students). Many students posted their introduction as a reply to my introduction. However, of the 122 replies to my post, I posted 30+ of those replies.

In short, I need to rethink this activity.

Implications

I wonder if the networks between student blog posts differs depending on when they posted to this discussion forum? Assuming that posting to this discussion forum on time is an indicator of engagement with the pedagogical intent?

If the aim behind an institutional learning analytics intervention is to improve learning and teaching, then perhaps there is no need for a complex, large scale enterprise (expensive) data warehouse project. Perhaps what is needed is the provision of simple – but currently invisible information/analysis – via a representation that is embedded within the context of learning and teaching and thus makes it easier for the pedagogical designer to combine the analytics with their knowledge of the pedagogical intent.

Answering the questions of what information/analysis and what representation is perhaps best understood by engaging and understanding existing practice.

@damoclarky needs to be encouraged to do some more writing and work on MAV and related ideas.

References

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing Pedagogical Action: Aligning Learning Analytics With Learning Design. American Behavioral Scientist, 57(10), 1439–1459. doi:10.1177/0002764213479367

#moodle Activity Viewer (MAV) and the promise for bricolage

I’ve spent the last few days – on and off – getting the Moodle Activity Viewer installed on my local Moodle instance. There were two main reasons for doing this

  1. Analyse how students were using my 2013 course sites.

    This will be the topic of later posts.

  2. Lay the foundation for exploring MAV as a platform for bricolage.

    This is the topic of this post.

Rationale

Over recent months I’ve heard various statements of the form “We know all there is to know about online learning and teaching”. Statements that reflect the perspective that the provision of quality learning and teaching at universities is a tame problem. It typically arises from experts – be they instructional designers or information technologists – and from people in “leadership” positions. Those in “leadership” positions seem increasingly convinced that leadership is the design of a single solution/vision to a problem and the successful implementation of that vision.

The problem is that by seeing “quality learning and teaching” as a tame problem they believe that it can be “solved in a linear fashion using straightfoward, reductionist, repeatable, sequential techniques”. As a consequence, you get the organisational decomposition of skills into different organisational units. This decomposition prevents connections between the disparate knowledge bases of technology, pedagogy, content and context. The difficulty (impossibility) of making these connections limits the capability of organisational learning and teaching to learn and improve.

What’s worse is that the “tame problem” perspective results in the adoption and perception of technologies (e.g. the LMS) as immovable. This results in the situation where if the technology doesn’t well support a particular pedagogy, then you better change the pedagogy because changing the technology is too hard. Again limiting the capability of organisational learning and teaching to learn and improve its practice. It also leads to the problem identified by Ciborra (2002)

..if every major player in the industry adopts the same or similar applications, any competitive advantage evaporates.

On a more personal level, all of this results in crappy systems that don’t actively help me improve the learning of my students.

For me, using technology to improve learning and teaching is a complex or wicked problem. The type of problem where lots of small scale, rapid experiments are the best way forward. The infrastructure underpinning MAV seems to be the best current foundation to enable this.

How MAV works

MAV is a plug-in for the Firefox plugin that communicates with a MAV server that provides access to a database. It enables the modification of a web page produced by Moodle. Currently it will modify a Moodle course page by adding a heatmap representing how particular groups of students have used the resources and activities on the course page.

It changes something that looks like this

Without heat map by David T Jones, on Flickr

Into something that looks like this

EDC3100 S2, 2013 - heat map by David T Jones, on Flickr

Now this is somewhat useful for a teacher wanting to understand how various aspects of a course site have been used (or not). It can be argued that this information is available via other means (e.g. Moodle’s activity report), but I’d suggest that the in-situ, colourful representation provided by MAV provides some additional affordances that the activity report doesn’t provide.

MAV does this using the following process

  1. I visit my course’s home page in Moodle.
  2. MAV recognises this as a Moodle course page and adds an “Activity Viewer” option to the Moodle settings.
  3. If I’ve turned MAV on, MAV then sends a request to the MAV server asking for how many students or clicks there have been on all of the links on the course page.
  4. The MAV server queries a copy of the Moodle database and sends the results back to MAV.
  5. MAV changes the background colours for all of the links (or it can change the size of the text) to represent usage. MAV also adds some text with the actual number of clicks or students.

But MAV’s real strength isn’t what it currently does, it’s how it could be used to support bricoloage.

It’s on my computer

The version of MAV that produced the above screen shots is running on my computer. The server is running on my computer. This means that I can write extensions to MAV to solve the problems I encounter when trying to support 300+ students in a course. If I come across a problem during semester, I currently have three options:

  1. engage in the heavy-weight processes associated with trying to get something changed in these systems (which probably won’t be able to be changed anyway); or
  2. implement some manual work around to solve the problem;

    e.g. create a zip file for each of the 60 assignments I marked and manually upload each one individually into the system.

  3. make do without.

For example, the pre-service teachers who take my course come from a range of sectors including early childhood, primary, middle years, secondary (content specialisations) and vocational education. The type of response I should give to a question can depend on the pre-service teacher’s sector. The Moodle discussion forum will tell me the name of the person who asked the question, but it doesn’t provide any other information. In fact, it can’t because information about a pre-service teacher’s sector is very specific to Bachelor of Education students and so is not part of the information from the university’s student records system that is inserted into Moodle.

It should be fairly easy to write a MAV extension that whenever it sees a student’s name, adds to the name the student’s sector. Perhaps even a mouse-over that shows a range of information about the student, perhaps including some personal annotations I’ve made about the student. Perhaps documenting (and reminding me of) the various unique complications that impinge on the lives of my students.

With MAV (and my capabilities), I can implement this modification without having to engage in the heavy-weight institutional processes. I can engage in bricolage.

This example probably doesn’t excite the learning theorists or instructional designers. It doesn’t offer any large change in the fundamental practice of pedagogy supported by an appropriately convoluted theoretical framework. It’s somewhat prosaic, simple, and only a very small change. But then such people don’t really get the concept of complex adaptive systems and bricolage (see below).

An aside on requirements gathering

I almost didn’t include the “pre-service teacher sector” example above. I found myself not being able to think of an example about how I might use MAV. This is not indicative of there not being a need for this sort of approach. It is indicative of limitations of human cognitive capabilities/memory and the stupidity of the assumptions underpinning traditional requirements gathering processes.

My difficulty in identifying example arises from the observation that I’m not currently teaching the course. Asking for requirements when I’m not engaged in an activity, is always going to result in significantly fewer and less detailed requirements than asking me while I’m engaged in the activity or actively observing me. And yet, how do organisations gather requirements for new systems? Months or years before people actually start using the system, they ask people, “What would you like to do with this system?”

The value of bricolage

Of course bricolage is always frowned upon by organisational folk. Bricolage is messy. It can lead to the ultimate evil in organisational IT – shadow systems.

But there is another perspective, again from Ciborra (2002)

If these approaches look rough compared to neat and tidy formal procedures, they are on the other hand highly situated: they tend to include an added element of ingenuity, experience, and skill belonging to the individual and their community (of practice) rather than to organizational systems. Finally, they all seem to share the same way of operating: small forces, tiny interventions, and on-the-fly add-ons lead, when performed skilfully and with close attention to the local context, to momentous consquences, unrelated to the speed and scope of the initial intervention. These modes of operation unfold in a dance that always includes the key aspects of localness and time (the ‘here and now’); modest intervention and large scale effects; on-the-fly appearance but deeply rooted in the personal and collective skill and experience

And drawing on research projects into Strategic Information Systems, Ciborra (2002) goes onto argue that

The capacity to integrate unique ideas and practical design solutions at the end-user level turns out to be more important than the adoption of structured approaches to systems development or industry analysis

and more directly for those who know the answers

All these cases recount the same tale: innovative, strategic applications of IT are not fully designed top-down or introduced in one shot; rather they are tried out through prototyping and tinkering. In contrast strategy formulation and design take place within pre-existing cognitive frames and institutional contexts that usually prevent designers and sponsors from seeing and exploiting the potential for innovation hidden in the artefacts….SISs (strategic information systems) emerge when early adopters are able to recognize, in use, some idiosyncratic features that were ignored, devalued, or simply unplanned.

References

Ciborra, C. (2002). The Labyrinths of Information: Challenging the Wisdom of Systems. Oxford, UK: Oxford University Press.