Category Archives: bim

Evaluating the use of blogs/reflective journals

The use of blogs in one of the courses I teach is now into it’s fourth semester. Well past time to do explore how it’s all going, evaluate some of the design decisions, and make some decisions about future developments. In preparation for that it’s time to look at some of the extant literature to look at findings and methods. The following is the first such summary and is focused on @spalm et al’s

Palmer, S., Holt, D., & Bray, S. (2008). The learning outcomes of an online reflective journal in engineering, 724–732.

Summary

Looks at the use of an “online reflective journal” (implemented using discussion forum in WebCT) in a 4th year engineering course. Combines a survey-based evaluation of student perceptions with “an analysis of student use of the journal…to investigate its contribution to unit learning outcomes”

Findings are

  • Most students understood the purpose and value the journal in their learning
  • Most read the entries of others and said this help their learning
  • Two most useful things about the journal
    • The need to continuously revise course material
    • Ability to check personal understanding against that of others
  • Least useful related to problems with using WebCT – the difficulty of the interface and “problems with CMS operation”.
  • Significant contributors to final mark
    • Prior academic performance
    • Number of journal postings
    • Mode of study

Thoughts

The quantitative nature of this is interesting as it’s something I need to do more of. If only to gain some experience of this approach and to tick the box (which is a great reason).

There is always going to be a bit of a “so what” issue with this type of thing as research. What really does a study in a single course reveal that’s new or easily transferable. But perhaps doing almost a replication addresses that somewhat.

Of course the real interest in doing this research is just to find out more about what’s going on in the course, its impact and what the students think so it can inform further development.

To do

  • Can I get a copy of the evaluation survey used?
  • Can I apply the evaluation survey to past students?
  • Do I need to get student permission to analyse the data around their use of their blogs and final results?

Comparison with EDC3100

My impression is that prior academic performance would be the most significant factor in EDC3100. Raising the question about how much value there is in what teachers do, if prior academic performance is a big contributing factor are we “failing” the weaker student?

Question: Could the students performance on the course be included in the evaluation survey in some form?

I think mode of study might play a role. My guess is that the online students would get see the value in the blogs more than some of the on-campus students.

Question: What impact does mode of study play in EDC3100?

Given that the blog posts were only marked based on number of posts, average word count and number of links it would be interesting to explore what impact that had on the final mark. Especially given the quote about “assessment” as a “strategic tool for creating student engagement”.

Question: Exactly what type of student engagement is the assessment of the blogs in EDC3100 creating?

Question: What patterns exist in the learning journal marks?

The journaling is contributes 5% of the total course mark for each of the 3 assignments. Is there a pattern in the marks for each assignment and the final outcome?

Question: Does the medium used make any difference?

In EDC3100 students are using their own blog hosted on their choice of external service. Very different from an LMS forum. Does this make a difference? Is it more their space?

Is it seen as too difficult or a waste of time creating a blog?

Question: Is the collection of technologies used to complete this task too difficult?

To get marks students have to create their blog (e.g. WordPress) and follow the blogs of others (Feedly, WordPress “follow” mechanism, WordPress reader etc). This leads to problems with creating links to posts. E.g. students link to the post in Feedly or the WordPress reader, not in the student’s blog.

But that said, the interface is “better” (subjective) and might be seen as more realistic – not a Uni tool, something broader.

Question: Does the time of posting make any difference? What are the different patterns of posting visible? Any patterns indicating task corruption?

Palmer et al (2008)’s journal is essentially a weekly task. In EDC3100 students need to post at least 3 posts a week to get full marks. There is no specific direction as to what or when to post. What patterns are there?

Question: How does the perception of reflection/journaling in the discipline impact thoughts?

Palmer et al (2008) describe how a work journal is a common practice for engineers. Hence doing this in the course can be linked to professional practice.

The same doesn’t apply to the teaching profession. While reflection is seen as important, there’s doesn’t appear to be the accepted practice of regularly keeping a work journal.

Making BIM ready for Moodle 2.6

The very nice folk from my institution’s ICT group warned me back in March that

I have started work on the moodle 2.6 upgrade that will be happening midyear and have come across some deprecation warning from BIM. Just giving you plenty of notice that an updated version will be needed before release.

That was just as my first use of BIM on the institution’s servers was getting underway. That’s gone reasonably well and it will be continuing (and hopefully expanding as I learn more about what’s required and possible with the approach) next semester, so I better get BIM playing nicely with 2.6. That’s what this post is reporting on.

BIM for Moodle 2.6 (and also 2.5) is available from the BIM plugin database entry and also from GitHub.

Get Moodle 2.6 running

Let’s get the latest version of Moodle 2.6 – 2.6.3 – and install that.

So that’s the first change. PHP setting for caching. Not that I’ll need that for testing. Looks like I can ignore it for now.

Get BIM installed

I’m doing this so irregularly now it’s good that I actually documented this last time.

That all appears to be working. Ahh, but I haven’t turned the debugging all the way up to annoying yet.

That’s better

get_context_instance() is deprecated, please use context_xxxx::instance() instead.

And about this stage it was always going to be time to….

Check the Moodle 2.6 release notes

The Moodle 2.6 release notes and then the developer notes. Nothing particularly related to this warning.

Do it manually

As outlined in this message it appears that this particular usage has been deprecated for a few versions. The deprecatedlib.php suggests this gets removed in 2.8.

So the changes I’m doing appear like this

#$context = get_context_instance( CONTEXT_COURSE, $course->id );
$context = context_course::instance( $course->id );

I can see this is needed in the following

  • ./coordinator/allocate_markers.php
  • ./coordinator/find_student.php
  • ./index.php **done?**
  • ./lib/groups.php
  • ./lib/locallib.php
  • ./marker/view.php
  • ./view.php – this one had actually been done earlier
    #$context = get_context_instance( CONTEXT_MODULE, $cm->id );
    $context = context_module::instance( $cm->id );

That all seems to be working.

Do a big test

Will back up a large BIM activity with a temp course from my Moodle 2.5 instance and restore it under Moodle 2.6.

Some more issues

print_container() is deprecated. Please use $OUTPUT->container() instead. Done

BIM testing and fixes

A journal of fixes and testing of BIM. Aim here is to address some minor issues with integration with my current institution’s Moodle instance thereby providing a minimum working version for installation. As per yesterday’s planning the hope is to make further changes based on this foundation.

Result is a slightly tweaked version released via Moodle contrib. This will be the foundation for some tweaks, though I can feel time slipping away.

Latest version of BIM and PostgreSQL

The institutional Moodle instance uses Postgresql. Thanks to playing with MAV I know have a version of Moodle running with Postgresql (aka fred). The plan here is to install BIM on that instance and test it

  1. What’s the latest BIM?

    MOODLE_25_STABLE is the latest, but MOODLE_24_STABLE is what I need for this work, institutional Moodle version still at 2.4.

  2. Install it on fred.

    Get the source

    git clone https://github.com/djplaner/moodle-mod_bim/
    mv moodle*bim bim
    cd bim
    git branch MOODLE_24_STABLE
    git pull origin MOODLE_24_STABLE

    Visit notifications as the admin user on fred and install of BIM successful.

    BIM not appearing in the list available in a course. A setting? No, there is an error? What error? Change ownership on the directory and all good.

  3. Do some basic tests with that version of BIM.
    • Create BIM activity in old EDC3100 course. – DONE
    • Do some work as administrator.
      • Register a blog – DONE
      • Create a question – DONE
    • Create some teaching staff – fred already has some details for users. – DONE

      Need to address the absence of the auth plugin – my laptop doesn’t have the institutional auth plugin, can I work around this?

      Need to create some new users.

      • examiner – david
      • marker – vick, rick
      • students – nerf, abe
  4. Do a BIM restore from the S2, 2013 version of BIM – this will be complex given usernames? – DONE

    This worked surprisingly well. Taken a bunch of data from real life S2, 2013 and placed it into the institutional version of the course and it’s worked all good.

  5. Check the known institutional problems
    • Bulk email – fixed.
    • User search.

      Stalling for some users. Works for others – having a registered feed may be a distinction?

      Having trouble identifying the cause. Wonder if it’s purely a Postgresql problem. Try with another version of Moodle with MySQL.

      Works, but generates an error about curl:$count in lib/filelib.php – there is a call to SimplePie. – suggesting that the problem isn’t Postgres, but the proxy configuration on the other Moodle server. Confirmed. This raises an issue with the timeout situation with curl (changed). But also about where this is being called – showing student details I imagine.

    • All teaching staff are coordinators – DONE

      Maybe due to how institutional roles are mapped to Moodle archetypes – examiner/teacher/moderator – editing teacher; tutor/non-editing teacher/marker – teacher.

      Vick Far – teacher (archetype editingteacher) – gets the coordinator view. Rick Nerf – marker (archetype teacher) – gets the marker view.

Some new issues

As doing the above testing, am adding issues into GitHub associated with a milestone. What follows is a record of dealing with those.

Undefined property warnings in locallib.php – 435 – Fixed.

Ugly about messages – Fixed. Raises some potential to offer better support to folk around BIM.

Share this with the world

These fixes need to be shared more broadly.

  • Back to github
  • Up to Moodle contrib

Done.

Identifying some immediate changes to BIM

I have until the 21st of February to get BIM tested and ready for installation into the institutional Moodle instance. The following is some initial planning of what I’d like to get done in that time frame. A list that will then need to be further whittled away to what I can get done in that time frame. There are three categories of changes

  1. Changes to better support the pedagogy I’m currently using.
  2. Changes from the BIM issues list.
  3. Changes to ensure correct functioning.

Better support the pedagogy

The pedagogy/learning design that informed the initial design of BIM is fairly limiting. The learning design/pedagogy I’m currently using isn’t directly supported by BIM. I found myself last year doing a range of programming kludges to get it to work. This won’t work in the second half of this year when a non-technical academic takes over as course coordinator. BIM better fitting the current learning design saves me time and enables other people to use this approach.

  • Allow more than one post to be allocated to a question

    Already had this as an issue. Would allow “questions” to be though of as modules (in EDC3100 speak) or time periods.

  • Allow student allocation of posts to a question.

    Mentioned in this issue (amongst other extensions).

  • The student interface would also need to be changed to handle the display of multiple posts to a question.
  • Have BIM generate statistics about length of posts, number of links and number of links to other student posts. Display this to the student and generate a CSV file for the marker.
  • Also seems to suggest some sort of auto mark generation based on the statistics.
  • Have a default allocation of posts to questions/topics based on the date OR just have it set to a particular value (i.e. all posts should now be allocated – by default – to Module 1).

The basic idea behind these changes is that students are required to make a number of posts per Module in the course (one module = about 3 weeks). Students mark for the posts for each module is based solely on the number of posts and that the posts meet a certain set of statistics (length of posts, links etc). These marks are added as part of the bigger assignment that is associated with the module.

Students need to be able to see their progress. Markers need to be able to access the statistics to mark assignments.

Changes required would likely include

  1. Changes to student interface
    1. Show question allocation as a row (not a column).
    2. Replace the “Status” column with a “Statistics” column that includes the number of words, links etc statistics
    3. What options exists with parsing HTML and extracting links in PHP and in Moodle.
    4. Show unallocated posts together in a separate set of rows – perhaps even a special form – where the questions allocation drop box is available.
  2. Changes to the coordinator interface

    1. Configuration option for multiple questions

      database change required

    2. Change the default allocation of posts.
    3. Some how deal with questions that aren’t questions.
    4. How to specify statistics for auto marking.
    5. What if any changes are required for the “Manage Marking” page.

      May not need it. As this page simply shows counts. With multiple posts to a question, the count should still work.

    6. What to do about the workflow and the idea of marked/suspended/released if posts aren’t being marked, simply analysed.
  3. Changes to the marker interface
    1. “Mark posts” will still need to be used to allocate posts (or should this feature be added to “View student details”) even though “marking” may not make sense.
    2. “Mark posts” cell with a question would need to show the number of posts for that question implying no way to mark directly. Or perhaps list each post? Needs thought here
    3. “Allocate posts” page will need to retain all question names in the “Choose one” drop box.
    4. “Allocate posts” page should also have an additional heading to group multiple posts to the one question — this may enable doing without the database change for the new configuration option.
    5. May need to add to “Allocate posts” a link to “mark this post” so that “Mark posts” page can point to a list of posts and one can be chosen for marking. Mmmm.
    6. One of these pages should have a link to export a CSV file containing marks for students against posts.

From the BIM issues list

The BIM source code is hosted on github and I’ve been using the associated issue list to record any ad hoc improvements/fixes. The following are the issues that would be nice

  • No response to find student – a bug, has it been fixed yet?

    Seems to specific to Moodle 2.5, so not directly applicable to the institutional context.

  • Warn of summary feeds

    Problem from last year, blogs configured to just showing the first few lines of posts, not the whole post.

  • A couple of issues about updating posts – allow students/markers to update posts stored in BIM (mostly to fix errors or recent changes).

Correct functioning

  • Check out any and all warnings being generated by BIM now.
  • Bulk email

    Used in a number of places. This is not working. Either on mootest or my box. A missing parameter. Not sure when this cropped up.

  • User search

    Search for a student within BIM isn’t working on mootest, it does work on my box. My initial guess is some SQL type queries in BIM that are MySQL specific.

  • All teaching staff are coordinators

    The distinction between coordinator and marker isn’t kicking in as it should.

Visualising posting frequency: BIM and EDC3100

The following describes an attempt to develop a visualisation of the frequency with which students in the course I teach posted to their blogs during two offerings of the course in 2013. A few reasons for doing this

  1. Confirm my gut feel that some students (for a variety of reasons) treated blogging pragmatically and only posted just before due dates.

    Only some evidence of this.

  2. Explore if there’s any visual correlation between this behaviour and their final grade in the course and other factors.

    Some suggestion of this.

  3. Help think about whether or not a visualisation like this might be something to include in the next version of BIM.

    That might be difficult given time-frames and other constraints, but I’ll be exploring how to create some of these manually and use them during the coming semester.

Cut to the chase (you may have to be patient for these visualisations to come up in your browser – and I can’t be sure they’ll work for everyone)

More description of what and how follows.

The first plan

A table. Columns represent days of the semester. Each row is a student. Colour the cells where a post occurs. Show the due dates by running a line through the table. Group the rows via various external factors including: GPA, result in the course, sector.

Google Chart’s magic table looks like a good tool for a first run. So here goes.

Test out magic tables

Well, the example code for Data::Google::Visualization::DataTable is truly broken. But using it will generate Javascript array that will slot straight into the MagicTable visualisation.

Doing it with BIM data

So bring in some of the BIM code, read the database and generate the data structure required by the MagicTable code and you get the following as a first stab (click on it to see a larger version). Each row of this figure represents an individual student blog. Each column represents a day of the semester. Any cell that is shaded a particular colour represents a day when some posts were added to the student’s blog.

The fish-eye view in the above is overing over one student who definitely seems to have left things to the last minute – 16 posts on the one day. A day that happens to be toward the end of the semester.

The prevalence of green squares earlier in the semester represents days with 6 to 8 posts.

If you want to be more interactive, you can see the full working HTML page here. The table itself has a size of 1403×1315.

What’s missing/wrong from this view?

  1. The first few columns don’t seem to be getting set correctly – NaN?
  2. Need more signposting of the various semester dates. e.g. when are the 3 assignments due? When’s the Professional Experience period? When are the semester holidays?

    On the HTML page it’s possible to see a pattern where there are fewer posts later in the semester. As it happens, the last 5 weeks of the teaching period for this semester included 2 weeks of holidays and 3 weeks of Professional Experience (where the students are out teaching in schools).

  3. Include some indication of student results – e.g. final grade, GPA, assignment result or even the result for the learning journal (the marks given for the blog).

    This example of a magic table uses the idea of column/row headers which could be useful in doing this.

Next step

If I move directly to generating the Javascript from Perl for magic table it appears I have a bit more control. I can set a row and column header so that you can see some detail about the column (e.g. the data) and row (e.g. GPA, student name, grade etc). I have that working. Now I need to get some extra student data into the database.

It is moments like this that I detest working in an environment where there is not decent data infrastructure. Having to waste time to fix up some of the data I’m working with.

Ok, now let’s sort by grade and here is the finished product. The page itself offers some more description of what is shown.

I don’t have the time to explore what’s going on here in detail. So, some quick observations, none of which tell you very much without further exploration

  • There is an early student with “no grade” (indicating a withdrawal from the course) that has apparently posted consistently throughout the semester.
  • All of the other “no grade” students don’t post after the 15th August (when the first assignment is due).
  • There’s a group of 5 students with the lowest passing grade who blogged consistently throughout the semester but started later than other groups of students.
  • There are a few students with the top grades who are quite obviously blogging just before the assignment due dates.

However, it does reinforce an existing opinion that the blogging in this course is not yet introduced sufficiently well for the students to get real value from it.

Semester 1

The semester 1 offering of the course had three times as many students and was the first time the blogs were used in this course. The semester 1 offering also included 3 groups of on-campus students as well as online students (semester 2 was only offered online). I’m thinking that the semester 1 map should show evidence of more “pragmatic” blogging.

Let’s find out.

A rough first run at the semester 1 view.

Some quick comments

  • The magic table widget breaks down with the a table this size. Difficult to see the row/column labels as the page scrolls.
  • At least for me, you can see the shape of the semester. Assignment 1 due – break. Assignment 2 due – Professional Experience (no blogging), then assignment 3 due.
  • THere’s at least one C student who appears to be blogging just about every day and often more than one post. What’s with that?
  • There’s a group of C students who appear to be very pragmatic.
  • There seems to be a general increase in posting regularity and intensity as you get to the HD students.

    At least this is how it looks to me.

Would be interesting to explore further. Does mode in semester 1 make a difference? Amongst many more.

What’s next?

Wonder how difficult this would be to incorporate into BIM in a usable way?

At this stage, I’m thinking I’ll produce a couple of these visualisations during the coming semester for students to think about where they are with their practice.

May also prove an interesting indication of how changes in the scaffolding and support of the blogging process works out.

The visual representation is interesting, but needs easier ways to manipulate and explore the data more. e.g. some representation of the size, content, number of links, and quality (?) of the blog posts etc.

Reflective Blogging as part of ICT Professional Development to Support Pedagogical Change

I am planning to do some more work on BIM in preparation for using it in teaching this year, including finishing some analysis of how the blogging went in last year’s two offerings.

As luck would have it, I skimmed one of my feeds and came across Prestridge (2014). What follows is a summary and some thoughts. It’s nice to be reading an open access journal paper after a few very closed off articles.

Aside: I am wondering whether or not in the new world order being someone that reads feeds and has students blog is become somewhat old fashioned.

Abstract

The abstract for Prestridge (2014) is

Reflection is considered an inherent part of teacher practice. However, when used within professional development activity, it is fraught with issues associated with teacher confidence and skill in reflective action. Coupled with anxiety generally associated with technological competency and understanding the nature of blogging, constructive reflection is difficult for teachers. This paper focuses on the reflective quality of school teachers’ blogs. It describes teachers’ perceptions and engagement in reflective activity as part of an ICT professional development program. Reflective entries are drawn from a series of blogs that are analysed qualitatively using Hatton and Smith’s (1995) three levels of reflection-on-action. The findings suggest that each level of reflective action plays a different role in enabling teachers to transform their ICT pedagogical beliefs and practices. Each role is defined and illustrated suggesting the value of such activity within ICT professional development, consequently reshaping what constitutes effective professional development in ICT.

This appears to be relevant to what I do as the course I teach is titled “ICTs and Pedagogy” and reflection through blogging is a key foundation to the pedagogy in the course. Of course, this appears to be focused on in-service, rather than pre-service teachers.

Introduction

In the Australian education context various government policies illustrate that ICTs are important. There’s a move to 1-to-1 student/computer ratios. However, “success with regard to technology integration has been based on how extensive or prominent the use of it has been in schools rather than on whether the teacher has been able to utilize it for ‘new’, ‘better’, or more ‘relevant’ learning outcomes (Moyle, 2010)” (Prestridge, 2014). Suggests a need to “reconceptualise both the intentions and approaches to professional development” if there’s going to be an ROI on this government investment and if we’re to help teachers deal with this.

PD is “an instrument to support change in teacher practice”. Long held view that PD should move from “up-skilling in the latest software” to a deeper approach that focuses on pedagogy and context rather than technology; building teachers’ confidence in change; development of teachers’ metacognitive skills; and, as a philosophical/revisioning of ICT in learning (references attached to each of these). References work by Fisher et al (2006) as requesting “a cultural change in the teaching profession”, the principles of which need to be “activiated within ICT professional development if we are going to move from retooling teachers to enabling them to transform their practices”.

Note: I wonder how well this academic call for a cultural change matches the perceptions of teachers and the organisations that employ them? I have a feeling that some/many teachers are likely to be more pragmatic. As for the organisations and the folk that run them….

And now onto the importance/relevance of reflection to this. Schon gets a mention. As does the action research spiral, teacher-as-researcher, inquiry based professional development, reflective action, Dewey. Leading to research suggesting “that reflection brings automatic calls in the improvement of teaching” and other work suggesting there’s a lack of substantive evidence.

This paper aims to investigate “the role of written reflection as a central process in a ‘hearts and mind’ approach to ICT professional development.

Note: The mix of plural and singular in “hearts and mind” is interesting/a typo in this era of standardised outcomes/curriculum and increasing corporatisation.

Methods to framing the research

Background on a broader ARC funded project that aims to develop “a transformative model of teacher ICT professional development”. With “one or two teachers” volunteering from each school it would appear to suffer the problem of pioneers versus the rest. Teachers engaged in classroom inquiries, in particular the “implementation of an ICT application in regard their pedagogical practices and student learning outcomes”. Supported through a local school leader, outside expert, online discussion forum and personal blogging.

Has a table that lists the inquiry questions of the 8 teachers. Questions range from “How can students be supported when creating an electronic picture book using the theme ‘Program Achieve”?” to “What strategies need to be employed to promote effective/productive ICT practices that encourage intellectual demand and recognise difference and diversity?”

Teachers were encouraged to blog after teaching. Provided with a framework for reflecting after teaching (5R framework). Weekly blog mandatory. School leaders asked to encourage blogging.

This work focuses on

  1. teachers’ perceived value of the reflective activity

    Data from teachers’ final interviews and reports analysed using constant comparative method

  2. the role of written reflection in enabling change in pedagogy

    Blog posts analysed with Hatton and Smith’s (1995) three types of writing: descriptive reflection; dialogic reflection; and, critical reflection.

Results

  • 1 teacher had consistent reflection of the year implementation
  • 4 teachers had spasmodic entries, mostly at the beginning
  • the other teachers writing could be seen as simply record keeping

Finds similar results in other use of reflective blogs and suggests that “teachers’ lack of understanding on how to reflect limits their reflective writing abilities”.

Note: Not a great result perhaps, but not entirely unexpected. Might get some idea of this from my students posts in 2013 later today.

Perceived value of reflective writing

Not surprisingly, the “consistent reflection” person liked blogging. Others didn’t.

A major theme on the value was “a lack of understanding on how to reflect”.

Note: I have a feeling this may be one factor for my students. Though I wonder how much pragmatism and especially how reflective blogging falls outside the realm of standard practice for many plays a role.


“What to write in the reflective blog and then what to do with these reflections were issues raised by the teachers”

Note: Raising the issue of BIM being better at providing “prompts” to students.

Ahh, a quote from a participant brings back the “realm of standard practice” issue

I think because this inquiry thing was such a different way of doing things I’ve ever done before, it took me a while to get fair dinkum about it. I still couldn’t get the blog…..that’s one positive that’s come out of it because if I were asked to do something like this again then I would do it much more readily.

Picks up on the idea of “reflection as description” through a number of quotes. An apparent lack of priority given to analysing on what had happened and going beyond description.

This is even though the teachers were given the 5R framework and a range of questions/prompts in project documentation and comments by the outside PD expert.

Note: Given this difficulty in understanding how to write reflectively, what impact does it have on the next part of the paper “examining the role of written reflections to identify how reflection supports teacher change in pedagogy”?

The role of reflective writing

The obvious “solution” is to focus on the 1 teacher who consistently blogged, generating the problem of a sample of 1.

The posts were analysed in chronological order. Emphasis on linking the type of reflection and the role it plays in “improving and or supporting teachers in transforming the beliefs and practices”

The most common type of reflection is descriptive, which really isn’t reflection. But this descriptive reflection “provides a leverage for dialogic reflection” which may or may not be pursued. As it turns out, generally not chosen. Only when a critical friend provides some additional prompting does it appear.

When it did occur, it helped shape the teacher’s pedagogical beliefs and practices. Descriptive reflection made conscious the connection between pedagogical beliefs and actual practice, but more as a justification.

Only spasmodic evidence of critical reflection.

Suggests that data supports the conclusion that there’s a developmental sequence to reflection. Start with descriptive and then the more demanding forms emerge.

The role played by each type of reflection in transforming pedagogical beliefs and practices

  1. Descriptive reflection – a connector, making conscious the links between pedagogical beliefs, current teaching practices, and student learning outcomes.
  2. Dialogical reflection – the shaper, where the connections were examined and explored, enabling transformation.
  3. Critical reflection – a positioner. Placing the role of teacher in the broader context and critical evaluate the role.

Conclusion

If how to reflect in written form is understood, then “reflective action plays a significant part in enabling them to change their pedagogical beliefs and practices”. Each type of reflection plays a different role.

A lack of guidance and support were found to affect reflective action.

References

Prestridge, S. J. (2014). Reflective Blogging as part of ICT Professional Development to Support Pedagogical Change. The Australian Journal of Teacher Education, 39(2).

Blogs, learning analytics, IRAC and BIM

In 2014 I am hoping to make some changes to BIM that will enhance the course I’ll be teaching. The hope is to leverage various learning analytics to enhance student learning. The following is an attempt to use the IRAC framework to think about what might be done. Essentially a bit of brainstorming about possible future development.

Each of the headings below link to the IRAC framework. First off the content and the purpose of this use of learning analytics is described. Then each of the four components of the IRAC framework – Information, Representation, Affordances and Change – are considered.

I’ve just learnt about the proceedings from the 3rd Workshop on Awareness and Reflection in Technology-Enhanced Learning, will need to read through that content for any additional insights.

Context

The course is a 3rd year course in a Bachelor of Education. It’s taken by folk hoping to become teachers at every level from prep, through Grade 12 and into the VET sector. The focus is the students be able to use Information and Communication Technologies to enhance/transform the learning of their students. During the course the students complete a three week practical in a school setting. The course is offered twice a year. The first offering has average around 300 students spread over three campuses and online. The second offering averages around 100 students all online. The students in the course are not necessarily all that ICT literate.

The students are required to maintain an individual blog that they use as a learning journal. The learning journal is intended to be used for capturing experiences, feelings and reflections. Contributions to the learning journal contribute 15% of the final mark. There is no formal marking of blog posts. Marking is done on the basis of the number of posts per week, the average word count, and the number of links to both external resources and blog posts from other students.

2013 was the first year the learning journal assessment was used. All 2013 student blog posts are archived in two instances of BIM. The plan is to use learning analytics to explore this data and test out various approaches that could be integrated into BIM and the course’s operation in 2014.

Purpose

At a high level, improve student learning while keeping the staff workload appropriate. Briefly, the pedagogy in the course is trying to encourage students’ self-regulation, reflection and building a PLN/making connections. I want the students to take ownership of their learning around ICTs and Pedagogy and for them to create and share a range of artefacts, insights and perhaps knowledge. The purpose that learning analytics can play in this is helping both the students achieve this and help the teaching staff support this.

Some high level aims for harnessing learning analytics.

  1. Provide students with some idea of how they are going and perhaps more importantly how to improve.
  2. Increase the diversity, quantity and quality of the connections between students and their posts and blogs.
  3. Allow teaching staff to identify who is struggling, who is doing well and who is in between and then help support staff in engaging appropriately.

A quick skim of 2013 course evaluation responses reveal some comments (emphasis added) from the semester 1 offering

The blog is a good idea to ensure the students are trying new ICTs during the course however the assessment was pointless. There was no real reason for us to be writing a certain amount of blogs per week. I found it a nuisance to maintain (on-campus student)

Probably the blogging was unnecessary but I still didn’t mind that. (on-campus student)

The blogs were very time consuming – and considering they were marked without being reviewed/marked then I am concerned that we could have done what ever we wanted! (on-campus student)

I also found blogging to be very beneficial in building my PLN (online student – most effective aspects of the course)

being forced to blog was actually great as it brought online students together as we shared resources and got to know each other. (online student – most effective aspects of the course)

the blog we had to keep, it had no purpose to it (online student – least effective aspect of the course)

the amount of blogs expected (online student – least effective aspect of the course)

The blogging, although I can see why we had to do it, I found it was hard to keep to the time frames as an online student (online student – least effective aspect of the course)

I don’t believe that I gained from blogging 3 times per week. I would rather have been assess on the quality of 1 blog per week and professional feedback that I could have provided to another student (rather than just links in the 3 blogs) (online student – least effective aspect of the course)

There were other comments on the blogs, common themes so far seem to be

  • The purpose of the blogs was non-existent to some, especially given that they weren’t marked based on quality by a human.
  • Blogs were potentially seen as more problematic in Semester 1 because of other issues with the course.

Change

Change is actually the last part of the IRAC acronym, but I’ll put it first. Mainly because it is the IRAC components that is least considered in learning analytics related projects (IMHO) and the one that I think is the most important.

In this case, I can see there needing to be three types of change considered: going outside of Moodle, using features inside of Moodle, and insidie BIM.

Outside Moodle

In short, thinking about, designing and implementing the type of changes to BIM and pedagogy outlined below is inherently a learning experience. I’m not smart enough to predict what is going to happen prior to implementation. I always gain insight when engaged in these activities that I want to leverage straight way into new approaches and new technological capabilities. i.e. I want to be able to make changes to BIM during the semester.

Not something I can do with the standard processes used for supporting a Universities institutional LMS. Hence the need to look at how I can do changes to BIM outside of Moodle and the institutional installation. In 2013, I did this via a kludge, essentially some Perl scripts and a version of Moodle/BIM running on my laptop.

Beyond the constraints of the institutional LMS processes, there’s the question of information and resources other than what is typically available to a Moodle module. Some examples include

  • Activity completion.

    Currently a small part of the 15% for the learning journal assessment in this course is based on students completing the activities for set weeks. This is in Moodle, but a module like BIM will typically not be able/expected to access this information.

    QUESTION: Can/how a module access information from other parts of a Moodle course site?

  • Student demographic and academic data.

    e.g. GPA of a student, how many times they’ve taken the course, might be used to help identify those at risk. Typically not information in Moodle.

  • Student dispositions.

    Data about students dispositions and self-regulation may be useful (see below) in providing advice. This would have to be gathered via surveys and would not be normally in Moodle.

  • Computationally heavy analytics.

    It is likely that a range of natural language processing and other potentially computationally heavy algorithms could be used to analyse student posts. Most enterprise IT folk are not going to want to run these algorithms on the same server as the institutional LMS.

All of this combined, means I’ll likely explore the use of LTI mentioned in this post from earlier in the year. i.e. use LTI to enable the version of BIM used in the course to be hosted on another server. A server only used for BIM in this course so that change can happen more rapidly.

In addition, that other server is also likely to run a range of other software for the computationally heavy analytics – rather than try and shoe-horn it into a Moodle module.

Inside Moodle

There’s a line of though – with which I agree – that learning analytics are most useful when supporting a specific learning design. The more specific, the more useful. This is in tension with the tendency of LMS tools to being generic. For example, much of what I’m talking about here moves BIM away from it’s original pedagogy of students answering questions to be marked by markers toward a more connectivist approach. Becoming more specific may limit the people who can use BIM. Not a big worry at the moment, but a consideration.

Moodle 2.0 has evolved somewhat in its ability to support change. For example, the introduction of renderers separates out the representation of BIM from the data and allows different themes to override a rendered. In theory, allowing other people to modify what is shown. However, the connection with a theme is potentially a bit limiting.

Task: Explore the concept of renderers more often.

Inside BIM

There is much that could be done to the structure of BIM to enable and support rapid development. e.g. Moodle is now supporting unit tests, BIM needs to move toward supporting this.

Information

To scaffold this look at the information that could be drawn upon, I’ll use the DAI acronym. i.e. the information to be used in learning analytics can be listed as

  • Data – raw data that is the starting point (e.g. blog posts for BIM).
  • Analysis – what method/algorithm is going to be used to analyse and transform the source information into….
  • Insight or perhaps Information – something that potentially reveals something new (e.g. how good is the reflection in this blog post)

Source

Information we currently have access to

  • All student blog posts from 2013.

    As part of the BIM database tables in the Moodle database.

  • The date and time when posts were made.
  • Student performance on assignments and in the course.

    Currently in a database in another non-Moodle assignment submission system. Pondering if this needs to move to the Moodle assignment submission system and thus the Moodle gradebook. But which raises a question..

    Question: Can/how would a module like BIM get access to Moodle gradebook data in the same course?

  • Some student demographic data.

    Currently as a CSV file manually downloaded from Peoplesoft by someone else. Includes age, postcode, sector, GPA.

  • Course and institution related dates.

    e.g. assignment due dates, semester start and end dates etc.

Information that we don’t have access to, but which might be useful

  • Comments on student blog posts.

    There’s no really standard way between different blogging engines of tracking and archiving the comments made on blog posts. So we don’t record those. Anecdotal observations suggest that many of the “connections” between students occur as comments. EduFeedr did some work around this.

  • Student perceptions of the learning journal assessment.

    Might be some mention in the 2013 course evaluation results.

    TASK: Take a look at the 2013 course evaluation results and see what mention is made.

  • Student dispositions and mindsets – e.g. this work.

Analysis

A very limited list of possible forms of analysis on the information we currently have

  • Link and social network analysis etc.

    Who is linking to who? etc.

  • Natural language processing, computational linguistics etc – which might open up possibilites such as

Combining the above with student demographic information and dispositions could also reveal interesting correlations and relationships.

I need to become more aware of what possible forms of analysis might exist. At the same time, the list of affordances (see below) may also suggest forms of analysis that are required.

Representation

Early suggestions for representation might include

  • Social network diagrams of various types.

    For students and teachers to see the structure and evolution of the social network of posts/blogs. e.g. this EduFeedr scenario

  • “My progress”

    Allow students to see a collection of stats about their blog and to see it in connection with others.

  • Student posting

The work reported in this paper on using badges gives on possibility for representation and also in terms of affordances for students to compare what they’re doing with others.

Affordances

The actual definition of affordances in the IRAC framwork – like the IRAC framework itself – is still in the early days of refinement. Here I’m going to use affordances as functionality that BIM might provide. Obviously influenced by the purpose from above.

  • Help students find interesting and relevant posts from other students.
  • Help students find interesting and relevant external links.
  • Allow students to see how “good” their blog is.
  • Show students how their blog compares to other students.

    There are reservations about this.

  • Allow all participants to get some idea of the important topics being discussed each week and over other time periods.
  • Show staff a progress bar/heat map/visualisation of some sort of student progress against expected milestones/questions.

    The EduFeedr progress visualisation below (click on it to see it bigger) is an inspiration.

  • Help staff to intervene and track interventions with all students.
  • Support staff in creating auto-marking approaches.

EduFeedr Progress

Measuring impact and improvement

If we ever get around to doing something in 2014, how will we know what’s changed? Alternatively, what might be useful to learn about the use of the student blogs in 2013?

Some possibilities

  • When did student post?

    Students were expected to have a number of posts each week, however, it was only assessed over a 3 or 4 week period.

    • How many students posted consistently each week and how many did the mad dash toward the end of the 3 or 4 week period?
    • Was there any correlation between when posts were made and the content of the posts, the students performance in the course, their GPA or anything else?
  • How did (if at all) did student posts change over the semester?
    • Is it possible to tell when holidays, professional experience, other assignments were due etc. from the student posts?
    • Did the emotions in posts change over semester?

      The course is quite heavy going. Especially in the first few weeks. I would expect some great nashing of teeth in the early weeks and perhaps in the leadup to assessment.

    • How did the connections between posts/students change over the semester?
  • Is it possible to develop indicators that might identify certain types of students/posts?
    • Indicators to identify students who are about to drop out?
    • Indicators to identify popular posts?
    • Indicators of students at all levels?

      e.g. what does a “good” student write about that an “ok” student writes about?

  • What were the most mentioned concepts during the semester?

To do

Some tasks left to do include, in no particular order

  • 2013 blog posts
    • Do some analysis of the 2013 blog posts.
    • Test out some of the planned analytics on these posts.
  • BIM
    • Explore the transition to renderers.
    • Explore unit tests.
    • Explore the “Moodle way” for assignments, marking, rubrics, outcomes etc.
    • Develop the “automated” marking feature.
    • Explore how the select “analytics” features will be identified.
  • LTI
    • Identify a good external hosting service.
    • Confirm that an LTI version of BIM will work with the course.
  • Purpose
    • Clarify exactly what pedagogical aims are going to be valuable.
    • Explore the self-regulated learning literature.
    • Look at the course evaluation responses from 2013 and see if there’s anything important to address.
    • Eventually identify a specific set of outcomes I want to work toward.
  • Information
    • Explore the various analysis methods that could be useful.
    • Explore how the analysis is best done with BIM, Moodle and PHP.
  • Representation
    • Explore how/if badges might be a possibility? USQ Moodle version and capabilities.
    • What PHP support is there for visualising social network diagrams?
  • Affordances
    • Get more into the literature around affordances, especially any work people have done on how to design affordances for learning/teaching.