Re-building results processing

It’s once again time to process final results for a course and return the final assignment. A process that involves

  1. Checking overall student results for a course, before returning the final assignment.
  2. Identifying all of the students who won’t have final results available by the required date.
  3. Analysing this offering’s performance across the course and comparing it with prior offerings.
  4. Returning the final assignment.
  5. Ensuring the overall results are entered appropriately into the student records system.
  6. Preparing a report for the examiners meeting.

There are three problems driving this

  1. What institutional processes/tools are provided to help with these tasks are far from user friendly.

    e.g. the greasemonkey script I wrote last year to help with task #5 above.

  2. There are no institutional processes/tools for some of these tasks.

    e.g. the only way to get all of the assignment marks into one place is to put them into the Moodle gradebook. But I believe if I do that then the students can see the marks. Given that I’m still moderating the last assignment marks them seeing the marks is not so good.

  3. The management of online assignment submission has changed this year, so some of prior workarounds are no longer viable.

What I used to do

Part of this activity is identifying what I’ve already got. It’s around 7/8 months since I last had to do this, so I don’t remember. I’m not sure the impact of this temporal distance is something that the designers of institutional systems and processes are fully cognisant of.

The old process appears to have been

  1. Extract individual CSV files for each assignment from EASE.
    The old online assignment system generated 2 separate CSV files. I needed both.
  2. Get another CSV file with Professional Experience results.
    Whether a students has passed their Professional Experience is determined by another part of the institution using a different system.
  3. Run a perl script which would

    1. Extract all the data from the CSV files.
    2. calculate the final results (for each student).
    3. Assign a grade to the student based on circumstances
      • FNP if at all assignments not submitted.
      • F regardless of mark if PE failed.
      • IM if PE mark has not yet been received
      • RN if assignment submitted but still being marked.
    4. Generate grade breakdowns for each campus and overall
    5. Output all that as a CSV file, including comments that I’d added
  4. Import that into Excel and do some further checking.

What I need to do

The process will be largely the same. The main difference is the source and format of the CSV files. The changes now are

  1. Assignment 1, Assignment 2 and PE marks are now in Moodle gradebook.
    The will all export nicely. Though it does use the long version of the USQ student number. May need to do a workaround for that.

    what sort of institution is silly enough to have at least 3 versions of the same student number?. Long with leading 0s. Long without leading 0s. Short!

  2. Assignment 3 mark will be in Moodle assignment activity.
    That will work. The grade is there.

    But yet, it uses a different version of the student number (long without leading 0s). But both do have email address, that might be the candidate for unique id. Or the script will just pre-pend leading 0s.

The other major difference is that neither of these CSV files include any mention of the campus or mode the students are using. Which prevents breaking results down. But thankfully I do have database table populated with this information.

Ahh, that’s right. I also have to change the PE mark for some students to -1 to indicate they were exempt. This is to discern students for whom there is no PE result yet, from those students who aren’t going on PE and should still pass.

My other problem is that the CSV file for A3 has a “status” field that includes new line characters. My poor little Perl CSV parsing module doesn’t handle that well.

It’s working well enough to help moderate A3 results. Time to do that.

Still to be done

  • Modify gradebook.csv to have -1 for PE for those students who will never go **** what about when it’s updated?
  • Add the campus calculations back in
  • Figure out how to handle the status field in Moodle assignment csv.
  • Have the script produce an Excel file, not a CSV file

Concrete lounges and why basketball players aren’t better hockey players

Assignment 1 for my course is due later today. 100+ students have submitted. Time to explore the Moodle assignment submission system and how to allocate assignments to markers.

So what?

Why would that be interesting to anyone not using Moodle? Well because…

Is elearning like teenage sex?

One explanation for the quality of “e-learning” is

We have taken our teachers who have been really effective at face-to-face teaching and asked them to also be experts in online teaching. Get real! That’s like asking a good basket baller to become a good hockey player. Yes it’s sport and yes you have a ball and competitors, but the rules are very different. And yes, if you’re a good sportsperson, chances are you can pick-up on being good at another code, but it will take time and quite a bit of training.

”Die Schuhe sind zu groß.” - ”The shoes by Jorbasa, on Flickr
Creative Commons Creative Commons Attribution-No Derivative Works 2.0 Generic License   by  Jorbasa 

That’s certainly part of the problem. But – to extend the analogy – the other part of the problem that I experience day to day is that universities are asking the good basketball players to play hockey with equipment that’s quite a few sizes too small and simply doesn’t help them play hockey, let alone learn how to play hockey.

This is not to say that the provision of the appropriate equipment is easy. It’s not. It’s incredibly difficult. A wicked problem.

The point is that the perspective (from the same post) is – in my experience – not the case at all

We already have all the tools we need to get our students engaged. Sure there will be new ones that come along from time to time that will do things a wee bit better, but for the time being we have plenty to make this happen.

As a teacher engaged with e-learning at a University, most of the technology provided is a concrete lounge.

Assignment submission

My current institution has this year moved away from a locally produced online assignment submission and management (OASM) system toward one embedded within the Moodle LMS. There’s apparently been some customisation of the standard Moodle OASM system, but it’s not clear just how much. I’ve already heard reports from other staff (with courses that have assignments due before mine) that allocation of assignments to makers is less than easy.

The following documents my attempts to do this and seeks to explore if the Moodle assignment submission system will be an example of the wrong size shoes for playing hockey.

I’m a hockey player

Background: I designed and implemented my first OASM system back in 1994/1995. From then through to about 2004/2005 I led the design and implementation of various generations of an OASM system and wrote the odd paper about it. I know a bit about these systems. I’m not a basketball player, I’m a hockey player.

Assigning some assignments – do it myself

Documentation by mray, on Flickr
Creative Commons Creative Commons Attribution-Share Alike 2.0 Generic License   by  mray 

First test, can I figure out how to do this via the interface. i.e. don’t read the “manual”.

Assuming the “Assignment administration” menu would offer some insight/affordances.

“Marker assignment quota” seems the most obvious option. A couple of observations

  • Apparently one of the students has some how been allocated the role of marker, she is appearing in the list of markers.

    My first question is obviously, “How the hell did that happen?”. The user is currently assigned to both the student and “general admin” roles. I don’t remember (even accidentally) making this change. Wonder how it happened?

  • This offers a choice of unlimited or a specific quota, but isn’t pre-populated with data already entered.

    i.e. to employ the markers to do this work, I had to negotiate with them how many they could mark and then specify that in the contract process. Having to re-enter this data in another system is a bit of a pain. I understand why it hasn’t been done. These are two very separate systems managed by very different parts of the institution. But if the shoe were too fit…..

Concrete lounge #1: Having to re-enter data already present in other systems.

View/grade all submissions

Next bet is to try the “View/grade all submissions” which shows a filterable list of all the submitted assignments and allows a number of operations to be done upon them. I’m assuming that “allocate marker” could be one of them.

Yep, “set allocated marker” is an option. Select the student(s) to allocate, select the menu option and hit “Go”. Brings up a page with those students listed and another drop down menu of markers. You chose the marker and hit “Save Changes”

Two potential problems with this

  1. Pre-allocation; and

    This does imply that you can only allocate markers to assignments that have already been submitted. I’ve got at least one marker who has a fixed group of assignments to mark. All located at a specific campus. In a perfect world I’d be able to pre-allocate the assignments from those students to the marker. Rather than have to wait until they are submitted and manually allocate them.

  2. Manually selecting individual students.

    Individual allocation is ok, but I would like to see at least two additional options. First, the allocation by group option available above. Second, a random (or perhaps specific) allocation of specified numbers. e.g. I have markers who will make 50 assignments, I’d like to automatically allocate them 50 assignments and have the system randomly allocate them. I’d rather not have to count to 50.

    Even better, it might be nice to say allocate them 50 assignments, but aim to achieve a balance of ability levels (perhaps based on GPA or some other indicator). Few things are more depressing than having to mark 50 low quality assignments. I assume there would be other allocation schemes people would like to apply.

Are there other options beyond this?

Grading options

Under the “grading option” drop down menu there is a “auto-allocate markers” option. But I wonder if it’s smart enough to handle variety. i.e. I need to ensure that one marker gets all the students at one campus, but can randomly allocate the remainder.

I don’t want to experiment with this option, just in case it goes ahead and auto-allocates straight away. So I’ll do a Google search for documentation. The search results are not that clear.

It appears that Moodle 2.6 added two related features – marking workload; and, marking allocation. These have to configured into the assignment activity. Did I do this? I did indeed. And this provides the functionality I’ve already identified.

So let’s just suck it and see. Good, it doesn’t do this straight away. It allows the options to

  1. allocate all unallocated, submitted submissions;
  2. allocate all unallocated submissions (including unsubmitted ones);

    Both these options are dong be specifying an “allocation batch size” and either doing an “Allocate” or “Simulate”. The simulate is a useful feature.

  3. copy allocations from another assignment.

Nothing here about allocating based on groups.

Filters and options

There are a collection of filters that can be applied, based on

  • # assignments per page;
  • assignment status;
  • marker allocation;
  • workflow status;

    A slight duplication to assignment status, but based on a different approach.

There’s nothing here about filtering based on groups. Is this because I haven’t configured there to be?

Group options in the settings

There is a “Group submission settings” section in the assignment settings. But most of this is based on the idea of students submitting assignments in groups. Not using groups to allocate assignments to markers.

No obvious options

I’m giving up. I can’t see from the system any obvious ways to allocate assignments easily via groups.

At this stage it appears that I will have to

  1. Manually allocate all students at one campus to their marker.
  2. Use the auto-allocate feature for the remaining students.
Edu Doggy by David T Jones, on Flickr

In theory, I could negotiate with the first marker to do an auto-allocate. But I think it important that he mark the assignments of his own students. Changing that preference would be the case of the tail wagging the dog.

Use the documentation

Before I do that, let’s see whether the documentation provided by the institution can offer any insight. It appears that this might be the solution

However, if the assignment activity has first been configured into groups, these can be manually assigned to a specified Marker.

I’m not entirely sure what this means. Let’s experiment with a dummy assignment and use the “common module settings” and the groups there.

First, the groupings don’t seem to be appropiate. No option to do it at campus level.

Okay, this appears to have added an additional option to filter which students/assignments are shown based on groups.

This would provide the option I need (a bit of a kludge), but the question is whether or not the setting can be changed on the fly – i.e. after students have started submitting?

The other question is what do (or will) the students actually see. I don’t believe there is actually an easy way for me to test this.

Let’s try making the change. Appears to be no problem with other assignments. I assume Moodle will warn of any horrible consequence? Famous last words? Logically there shouldn’t be a problem, but….

Change made, but there is no difference in the display. The option to select just the students in a particular group does not appear. Perhaps it can’t be changed once submissions have been made.

Concrete Lounge #2: No apparent way to filter assignments/students by groups

Group membership is stored independently of assignment submission in the Moodle database. It should be possible to offer a “Group filter” – perhaps even one dependent on the “grouping” – as a way to modify the viewing of all submissions.

Looks like I’ll have to do this manually.

Documentation at the wrong abstraction layer

Concrete Lounge #3: The local help documentation (like most help documentation) is written at the level of functionality. It describes the interface and what each interface element does. It isn’t organised at the level of “activity type” i.e. the level of the users.

i.e. I have a certain model of how I want to manage the submission, allocation, and marking of assignments. That’s what I know. That’s where I am. Documentation that started at this level by describing a list of different models of using assignment submission and then describing how to configure the Moodle assignment submission system to implement this model would be more useful (and much more difficult to write).

Better yet. Would be an assignment submission system that would present a list of different models, briefly explain each one, allow me to choose which I want, and then set up the system to fulfill that model.

i.e. the system actually fit what I wanted to do, rather than required me to engage in explorations how to figure out if and how I could translate the functionality of the system into what I wanted to do.

Sorry, but the tools we have available at the moment aren’t quite ready to help basketball players become better hockey players.

Update

As per the comment below I missed an option to flick. That’s done and I can see the groups and make use of those. So here’s what I did

  1. Allocate unsubmitted from campus X to the marker;

    Set filter to the tutorial group I need and filter for “unsubmitted”. This is so that if they submit, they will automatically appear on the marker’s list.

  2. Allocate submitted from campus X to the marker;
  3. Auto-allocate the remaining submitted to markers;

    Priority is given to those submitted.

  4. Drop the allocation for campus X marker

    Problem: the campus X marker was originally allocated 22 students to make. But one has dropped out. Meaning when I do an auto-allocate (simulation) he gets allocated another marker.

    I also have to make sure that the “student marker” has an allocation of 0.

  5. Do the auto-allocation again.

Now all I need do is to figure out how much advice the markers will need to download, mark and resubmit the their allocated assignments.

What advice is there?

Hard to explore this myself as I don’t know how much my view of the system is the same as what a marker would see.

The “Download all submissions” option gets them all, not just the ones I’ve allocated.

Appears that the “view all”, play with filters, and then download is the way to go. I assume that the markers won’t need have the “Marker filter” to play with.

I wonder if the organisation has given any advice specific to markers. Of course, the “portal” I can access links to various staff support sites won’t let me login on my current browser. Nor on chrome.

Oh dear, the “portal” still has a large explicit link to the old assignment submission system.

Now begins the traipsing through various sites to figure out where it is.

A couple of red herrings and finally found the document I had been using (not only was it hidden away in the object repository, I had to login again to access it) and it confirms my suspicion. Downloading will be fairly simple for markers – once they find the right place and buttons to push.

But there doesn’t appear to be any specific file/resource that can be sent to markers. It appears that I’ll have to create my own (just like every other person in charge of a course with multiple markers). Of course, the other option is that I’ve missed the presence of this other document entirely.

It appears a cut down version of the larger document was circulated. Found this out via personal networks and Twitter, rather than via other means. The smaller document had been circulated earlier via email, but finding it in my Inbox……

The documentation is very generic. I’ll update it and include a direct link to the specific assignment.

What is downloaded?

A zip file with all student submissions in a single directory. Wonder how it works if the students are submitting multiple files? Does it put each student’s submission into separate directories then?

Specifying moderation samples

In terms of moderation, my practice is to specify to the marker at least 3 assignments that they should mark first. These are sent back to me ASAP for moderation and any advice on marking. The aim is that the 3 assignment represent a spectrum of abilities based on GPA typically: a 6+ GPA, a high 4/5 GPA, and a sub 4 GPA.

As in the past, this information isn’t part of the OASM system. So I have to do it manually via a spreadsheet. However, in the past the OASM system did provide a spreadsheet of students allocated to a marker. This enable the use of Excel formulas to find some samples. Doesn’t appear possible to do this via Moodle.

Luckily the “more student details” popup described here lets me click on a link in the list of students and find out the students GPA (amongst other things).

Concrete lounge: Can’t easily allocate sample marking based on student GPA (or other means) in part because can’t see how to expert students allocated to a marker to a spreadsheet.

Contacting the non-submits

Another task at this time is to approach the students who have yet to submit the assignment and see what’s going on. Some of these will have extensions, some won’t. Is there an option to show those students who have not submitted, but haven’t received extensions?

Doesn’t appear to be.

Concrete lounge: Can’t see how to list those students who have not yet submitted the assignment, but who haven’t been granted an extension.

The options appear to be scrolling through the list of almost 50 students and manually identifying those without extensions. But even when I do that, what can I do? Can I send those students an email message?

Doesn’t appear to be possible.

Concrete lounge: Unable to send group (or personalised) emails to students who have not yet submitted the assignment.

Wouldn’t be too hard to write a Greasemonkey script that extracts the email addresses of the students without extensions.

  • name in c2 (cell 2)
  • email address is in C4
  • an extension is indicated by a div with class extensiondate in c5

But that would require a bit of extra work. I do have some Perl scripts that I use for web scraping that could be more easily converted, but not as shareable. Script written and email sent.

Concrete lounge: Unable to use filters to identify students who have not submitted the assignment AND do not have an extension

Uploading results and marked files

Uploading the marked files seems fairly straight forward, as long as the same filenames are retained.

Question remains about how to upload the marks. The OASM system won’t be smart enough to extract the results from the uploaded files.

Oh dear, it appears that results need to be added manually for each student. That’s a bugger if you’re a casual marker employed to make 50 assignments. Beyond the time and workload implications, there’s the problem of human error, especially with a hugely repetitive manual process.

Correction, I need to enable the “offline grading worksheet” option. Yep, that adds “download grading worksheet” (and upload) to the options.

How to help improve the Moodle book module

As mentioned in the last post there are limitations in the Moodle book module that are acting as a concrete lounge. The following documents an attempt to help improve the module.

In the end, I’ve made some serious steps to contributing a bit of code to a standard plugin (as I write this, I’m imagining how experienced Moodle folk will shudder at the many misuses of Moodle terms in the following).

Update: The “bug” in the Book module reported below is not a bug. There’s actually functionality in the Book module to support this, it’s just not obvious. More explanation is available.

Task and context

As described in the last post the context is a course website hosted in the Moodle LMS. A course that uses quite heavily the Moodle book module. The module has import function that is meant to allow you to import a collection of HTML documents into a book.

Concrete lounge

The problems with the import process include

  1. Titles for imported titles are 100% from the HTML filenames.
    e.g. call your file “Diigo tool disappears_sub.html” and the chapter title will be “Diigo tool disappears_sub.html”. Rather than “Diigo tool disappears”.
    The _sub is required by the import function to indicate a sub-chapter. The .html is to indicate a HTML file.
  2. The order of imported chapters is based on the order in the file system.
    e.g. If I have the following files to import

    dj:001 david$ ls *.html
    A blog.html
    Diigo tool disappears_sub.html
    Diigo.html
    The tools we’ll be using.html
    Twitter.html

    Regardless of the order in which I add these files to the zip file (import is via a zip file), the order that they will appear as chapters in the book is dependent on the order the file system of the Moodle computer creates.
    Even if I create the zip file in the right order

    dj:001 david$ unzip -t import.zip
    Archive: import.zip
    testing: The tools we’ll be using.html OK
    testing: Diigo.html OK
    testing: Diigo tool disappears_sub.html OK
    testing: A blog.html OK
    testing: Twitter.html OK
    No errors detected in compressed data of import.zip.

    The first chapter will be “A blog.html”

  3. There is not option to remove all existing chapters before import.
    There are situations where I’m trying to replace the entire book with the import. Currently I have to create a brand-new book resource, or manually remove all the chapters in the current book.
  4. No method for linking to a chapter prior to import.
    If I create the collection of chapters elsewhere, there will be times when I wish to create links between those chapters. Currently there is apparently no capacity to do this.

Bricolage

Here’s what I’ve done so far to address this problem with the resources available.

Process to make changes to Moodle core

This seems to be the process I need to follow for the above which translates into

  1. Is there a tracker issue?
    Via various trails I end up at the book page/section in tracker. Raising the question which versions of the book Module am I dealing with?
    Time to track through the book issues in tracker. Those related appear to be

    • support for re-importing chapters
      Close to what I’m working on. But it’s old. Created in 2008.
    • Book HTML zip incorrect URL
      Picks up on the issue with links between chapters that have been imported. This appears to have been fixed and should be in from 2.5 onwards. Need to check this (NOTE: I haven’t actually tried this process yet).

    Nothing apparent.

    So as a user I can report the bug/make a feature request and then contribute by making a fix. So let’s create an issue in tracker. What type of issue should it be?

    • bug – A problem which impairs or prevents the functions of the product.
    • improvement – An improvement or enhancement to an existing feature or task.

    Either could apply. I’ll be positive and call it an improvement. Issue created.

  2. Which branches should the fix be made on?
    The advice is

    Bugs should normally be fixed on all the supported stable branches that are affected. New features should just go into master, but sometimes minor enhancements are made on the most recent stable branch.

    So where does this fit? I assume a bug or minor enhancement. The other factor is that if I make this change on the version that my institution is using, that will increase the chances that I might get to use it this semester (which are very slim). Of course, it’s difficult to remember/discover which version of Moodle we’re using. I think perhaps 2.7 for now. Which seems to the other supported release.

  3. Develop the change using git
    Prior work contains an example of how to get a version of the Moodle source using git and start making changes. Will start with that.

    My issue number is MDL-49128

    The process I’ll use is

    • Fork the branch of Moodle I want to work on MOODLE_??_STABLE

      dj: david$ history | grep git
      git clone git://git.moodle.org/moodle.git
      git clone git://github.com/djplaner/moodle.git
      git branch -a # what are the available branches
      git remote add upstream git://git.moodle.org/moodle.git # connect upstream
      git checkout MOODLE_27_STABLE

    • Create the branch

      git checkout -b MDL-49128-book-import-fixes origin/master

    • Make changes – see below
    • Testing process
      • zipfiles.zip
        Has the following files that should be displayed in this order

        1. The tools we’ll be using.html
        2. Diigo.html
        3. Diigo tool disappears_sub.html (this is a sub chapter)
        4. A blog.html
        5. Twitter.html

        Without either fix the chapters will start with A blog.html, Diigo.html etc. through to Twitter.html.
        With the both fixes, the order of chapters should match the numbered list above. Also, the chapter names should not include .html or _sub

      • zipdirs.zip
        Has essentially the same chapters, however, they are implemented as folders containing a single HTML.
    • Commit the changes to my repo
      This wasn’t as straight forward as I thought. Ended up with

      git remote set-url origin https://github.com/djplaner/moodle.git
      git push origin MDL-49128-book-import-fixes

      Which ends up creating this nice little summary on github.

    • Ask the Moodle devleopers to review it
      Updated the issue in the tracker, time to wait and see what I got wrong.

Making changes

I’ve been using the zipfiles approach. A zip file with a collection of HTML files, each file equals one chapter. This is fixed based on previous work.

There is also a zipfolders approach. A zip file of folders. Where each folder equals one chapter.

The order fix is working on zip folders. However, with the chapter names, I’m now getting both the chapter name and the file name. So The tools well be using/The tools well be using. Should the filename or the folder name be used?

I’m assuming the folder name. The book module import code appears to support this. For example when reading HTML files it is looking for default files in the folders. i.e. Default.htm and index.htm. Suggesting that it isn’t assuming that these have the names of the chapters.

That appears to be working.

Factors

I have to think that anyone who had used the import function in action – like I tried to do – would have been aware of the limitations of the import function. It just doesn’t seem designed to be useful. Thus it seems logical to suspect that maybe the original designer wasn’t a user. They weren’t solving their own (or someone close to them) problem. Just implementing a feature. Hence they weren’t in a position to know how limited it was.

This distance between implementing the function and using the function seems to be a common feature of concrete lounges. Anyone who has to use a concrete lounge for any length of time is going to be aware of the pain and want to change it.

The fact that this particular bit of pain has stood for so long, has to say something. Perhaps it’s just at the book module is just an unloved child? But it is part of Moodle core now, so it has to be perceived to be useful for some.