The story of BIM – the slow expansion of BAM

Episode 3 of the story of BIM

A descent into Limbo

The initial development and use of BAM was in a single course that finished around November, 2006. This was the last course I was to teach. By early 2007 I had applied for and been selected as the new Head of E-Learning and Materials Development at the same institution. While still an academic role, I would no longer be teaching. Instead I would be the supervisor of a group of 10-20 staff who were responsible for various tasks including curriculum design and desk top publishing. It wasn’t until the middle of 2007 that I became responsible for the support of CQU’s official LMS (Blackboard).

2007 was the start of a significant period of uncertainty. By the time I took over e-learning the Director who attracted me to the position had not had her contract renewed. While I was the designer of Webfuse and very critical of the nature and implementation of traditional LMS (like Blackboard), I found myself in the position of being responsible for Blackboard, but not Webfuse. At around the same time a new Director of information technology was appointed and there was a major re-alignment of IT. i.e. All IT should be the responsibility of the IT division. Various other factors contributed to on-going uncertainty about how to achieve anything related to e-learning. Perhaps the biggest was the perception that the institution was in financial peril and subsequent decisions not to renew contracts.

CHALLENGE #13 A context of increasing uncertainty and reducing funding does not contribute well to the development of innovative pedagogy.

In addition, there were significant shortcomings in the existing processes and outcomes underpinning how materials development and e-learning were currently being done. Not to mention, that after many years absence, it was necessary to figure out how curriculum designers could once again be incorporated into the institution. It was thought that this should be the focus of what I was doing, not on further developments of BAM.

To give some idea of the level of uncertainty. I was employed as the Head of E-learning and Materials Development. But the unit I was in charge of was called the Curriculum Design and Development Unit (CDDU). As stated above, CDDU did not initially include responsibility for e-learning. In addition, CDDU had no official input into the institution’s learning and teaching grants process.

In this context, there were no plans to complete development of BAM. Instead, ad hoc support would be provided to people who wished to use BAM with the understanding that once the situation became more certain decisions could be made about what to do and what not to do. This never really happened.

Subsequent use of BAM, 2006-2009

The following table gives a breakdown of BAM usage by each term. It summarises the number of courses using BAM, the number of staff and students in those courses and the number of blog posts managed by BAM. In summary, during this period 69 staff used BAM to manage 19969 blog posts made by 2789 students.

Usage statistics of BAM: T3, 2006 to T3, 2009
Period # Courses # Staff # Students # Blog posts
T3, 2006 1 10 138 1009
T1, 2007 1 10 185 1561
T2, 2007 3 5 188 1301
T3, 2007 1 5 83 483
T1, 2008 3 8 277 1977
T2, 2008 5 15 247 1596
T3, 2008 1 9 202 1474
T1, 2009 4 22 596 4434
T2, 2009 5 26 639 3443
T3, 2009 3 11 139 519

During this period, there was essentially no publicity for BAM. The staff who chose to use BAM in their courses fell into the following categories:

  • They were lumbered with BAM because they were employed to run a course that had already been set up to use BAM.
  • Were guided to the use of BAM by someone working within CDDU.
  • Based on an earlier experience of BAM, they wanted to use BAM in another course.

The course for which BAM was initially developed was particularly was taught be a different person in each of the following 6 offerings. Only on the 7th subsequent offering was it taught by someone who had taught the course while BAM was being used. In practice, this meant that each of the people teaching the course were using the assessment designed by the previous person. Not assessment they had designed. It was on this 7th offering, that the decision was made to drop BAM from the course.

The limited consistency in responsibility for the course makes it difficult to develop a sense of ownership of the course. This can limit full engagement with the intent of innovative pedagogy. The courses where BAM has been most successful, has been where the coordinator/designer of the course has been instrumental in decision making. “Ownership” of a course and the decisions around its design are increasingly difficult in a context where courses are offered multiple times a year and where, increasingly, teaching staff are casuals. Especially with the application of innovative pedagogies which are challenging the status quo. It often takes time to become familiar with the innovation and the ramifications of the changes it creates within the context of a course.

CHALLENGE #14 A context where there is limited consistency in responsibility for a course, make it difficult to learn about the problems and evolve an innovative pedagogy.

From the start an eventual interest of mine in developing BAM was to explore more social applications of the blog technology. Especially, investigating applications that lean more to co-operation than collaboration as suggested by Stephen Downes. However, once I no longer was responsible for the course, I no longer had the “power” to make that decision about a course. It was up to the coordinators.

CHALLENGE #15 Innovative pedagogy is limited by the conceptions, beliefs, desires and aims of the coordinators of a course.

The hobbled BAM

A common theme underlying the design of BAM was that in its first use I would be the coordinator and I had command line access to the server on which BAM was running. Not to mention the expertise to use it. That meant that it wasn’t necessary to develop a web interface for the coordinator/management tasks for BAM. Examples of these tasks included: configuring a course to use BAM, releasing marked posts to students, running copy detection over student posts, changing students registered blogs and merging BAM results with the institution’s end-of-term results processing system.

Since the limbo period resulted in no changes to BAM, coordinators of the above courses had to make do with BAM as is. There were unable to perform these tasks. Instead they had to ask me to perform them on the command line. This introduced delays.

??What’s the challenge here, there is one, how to phrase it?

Most of the courses that were using BAM during this stage were being taught across most of the institution’s 9 campuses (spread across the eastern seaboard of Australia) and by distance education students. The institution and each of its 9 campuses have an infrastructure and set of processes around offering support for information technology ranging from desktop support through to e-learning and beyond. This infrastructure did not recognise BAM as a centrally supported IT system. It could also be argued that few people within this infrastructure knew anything of the centrally supported IT system on which BAM was based. This lack of knowledge led to some confusion when teaching staff new to BAM asked their local IT folk about problems they were experiencing.

It is difficult to see how an IT infrastructure across multiple campuses, run under different regimes could fully understand and respond to all applications of innovative blended learning. How many courses does an innovation need to run across before it become cost effective for the IT infrastructure to disseminate training about that innovation to its IT support staff? 10, 20, 300, 1000?

CHALLENGE #16 Innovative applications of blended learning often operate at a scale that does not match what is required by an enterprise IT approach to support.

In the case of BAM, this was somewhat increased by the reliance on external blogs and RSS. The majority of staff and students had little familiarity with these technologies and vanishingly small numbers had actually create a blog prior to this course. What about the net generation. Aren’t students already meant to be familiar with the Internet and social media?

CHALLENGE #17 The vaunted net generation doesn’t quite appear to have arrived.

It’s extra work

All of the staff who have used BAM comment on the extra amount of work it involves. This perception arises from a number of different sources, including:

  • Blogs, bams and reflective journals are all new to most of these staff and their students. Novelty means more effort to become familiar and find out the strategies and tactics to work around the flaws.
  • The design of BAM itself was not always the best and added to the amount of work.
  • Over reliance on old mindsets or a misunderstanding of the rationale behind the use of BAM.

There appeared to be a correlation between how negative a staff member was about BAM and how strongly they viewed BAM simply as another assignment to be marked. A significant number of the staff using BAM are employed as casual academic staff. This means that they are given a fixed number of hours to lecture, tutor and mark based on the number of students they are responsible for and the subsequent number of tutes. For example, if 25 students fit in a tute and you have 50 students you are paid for 2 tutes per week. Your payment for those 2 tutes includes payment to do whatever marking of assignments you are expected to do.

Courses using BAM almost always included BAM usage as an assignment that the staff had to mark. In the case of COIS20025 this was an extra assignment on top of the normal two assignments. This mean staff had to mark an extra assignment without any extra pay!

CHALLENGE #18: Fixed methods for the calculation of workload and subsequent payment to causal teaching staff limit the flexibility needed for innovative blended learning pedagogy.

Not surprisingly, these staff saw BAM as yet another assignment, and one they weren’t being marked for.

On the other hand, the coordinators who pushed hard for BAM saw BAM as an instrument to improve learning and teaching. To address fundamental problems with learning in courses. To make teaching more student centered. To make student thinking more visible in order to allow staff to help students be aware of any limitations before submitting assessment that would be marked.

The coordinators choosing to adopt BAM typically did so from a “visionary” or “quality” perspective. Other staff saw it through a more “pragmatic” perspective and consequently there experience was very different. Many, if not most, students also adopted this more “pragmatic” perspective and saw BAM as yet another assignment in which they applied there value logic. i.e. how much is the assignment worth in raw percentage terms, what do I need to do to get the percentage I need.

It was common for similarly pragmatic students to ask why they need to do this on a blog. Couldn’t they just do it on paper or a Word document? Why do I have to learn this new technology? Why all this extra work?

CHALLENGE #19 Innovative pedagogies are perceived in very different ways. The differences in perceptions influence how people engage with them.

Late enrolling students and student signatures – Indicators project?

Just floating another idea for a research project around the Indicators project.

The spark

In reflecting back on the origins of BIM, I generated the following stats about late enrollments

73% of the students in this offering enrolled in the course after term had started. In fact, 21% of the students were enrolled on the last day of week 2 of term. Supposedly the last day students could enroll. A further 9% of students were enrolled after week 2.

This was in 2005. I’ve since had confirmation that the problem continues to exist from another staff member. Along with the suggestion that in one course, all of the students who enrolled late, failed the course. Correction: I’ve just been corrected. what actually happened was that “all of the students who failed had enrolled late”. i.e. enrolling late – in this course – is a good predictor that you will fail the course.

I’ve just checked the 2010 offering of the 2005 course. 48.5% of students enrolled after the term had started. On the positive side, none of them have enrolled late. For another course I’m interested in this term, the figure was 52%

Student signatures

One of the “patterns” we have talked about is the idea of developing a “student signature” of LMS usage. i.e. a pictorial (or perhaps mathematical) way to represent how an individual student uses the LMS. The idea is that there might be differences in how they use the LMS. Something that might be used as a predictor. i.e. if you see pattern X in week 4 for a student, there might be a strong likelihood that they will do Y.

Amazing the serendipity in the blogosphere. The day I posted this, but after I’d posted it, I see David Wiley’s post about some of the work they are doing. The waterfall visualisation he shares could form the basis for one type of student signature, and could be interesting in looking for patterns around the topic of late enrolling students. However, as Mark Smithers asks in a comment, defining how you calculate the amount of time a student spends on assessable activities might be difficult. Initially, we might have to rely on site visits as a proxy (though I’m getting uncomfortable with this use of proxies).

What might be done

It would be interesting to:

  • See if late enrolling students have a different signature.
  • Whether they are more likely to achieve a certain outcome.
  • Find out what they are feeling when the start a course late.
  • Why they are starting a course late.
  • If this significant late enrollment can’t be stopped, what strategies can be adopted to help the students get going.

The story of BIM – Development of BAM

This is the second episode in the continuing story of BIM. This episode picks up the story late in the first half of 2006. The initial foray into using individual student blogs to encourage reflection and make student learning more visible was bumbling along. It was now time to think about what I’d be teaching the following term.

Introducing COIS20025, Systems Development Overview

Since 1990 I had been teaching within the information technology program at CQU. For various reasons I was no longer happy teaching within that program, one of the reasons is that through my work in e-learning I’d realised that there was a lot more to technology systems, than technology. You can make a beautiful bit of technology, but if it involves change on the part of the users, they won’t use it they way you intended (if at all). As a result, from the second half of 2006 I was moving into the information systems program. Theoretically, information systems is interested in the combination of software and otherware (i.e. the people, organisations etc.).

My first teaching task within the information systems program was the course COIS20025, Systems Development Overview. It was a Masters level course looking intended to give an overview of how information systems were developed (somewhat surprisingly – though not really – it tended to focus on the technology and methods and very little on the otherware). Traditionally the course was taught across multiple campuses by 10+ teaching staff and 200+ students. I would be the course coordinator responsible for managing all this and in addition teaching the students at the Rockhampton campus and those studying by distance education.

Check what early emails I have about this course. And my move to the information systems school. Give the background here on the course.

Thinking about COIS20025

By early 2006 I was starting to look at and think about what I might have to do with COIS20025. History showed that there were some concerns about the course. For example, a previous coordinator had received a “please explain” about why there was only 1 student getting a HD from 384. The response from the coordinator explained this drawing on reasons that were widely recognised by those at the coal face. i.e. that the majority of the students came from a non-English speaking background (NESB), most had little familiarity with technology, COIS20025 was taken in the first term they were in Australia getting to grips with a foreign culture and these problems were made worse by the fact that 73% of the students in this offering enrolled in the course after term had started. In fact, 21% of the students were enrolled on the last day of week 2 of term. Supposedly the last day students could enroll. A further 9% of students were enrolled after week 2.

Some comments from the previous coordinator

there is nothing in the course itself that makes me believe that it is too difficult for Post-Graduate students to manage…I can not see what else can be done to improve the results in COIS20024 and COIS20025 with the exception of capping the number of students we enrol and haivng a more stringent entry requirement for English Language Skills.

It could also be observed that the methods used in the course differed little from traditional methods from the glory (and probably mythical) days of higher education of motivated students.

This particular tension between teaching staff and the management of teaching staff is not new or specific to this course. My first experience of it was the late 1990s and have seen it in numerous courses. To this day I see no evidence of significant and successful changes to address this problem.

CHALLENGE #6: The increasing complexity of the teaching context within some Australian universities has resulted some significant tension and differences of opinion between management and teaching staff. Many of these remain unresolved.

While the changing nature of the students and the context suggests a need to change toward more innovative methods that engage the students, that address the limited performance of students. Tutty et al (2008) capture the more common approach

The solution to the high failure rate was to change the assessment to satisfy the institutional requirements of satisfied students and reasonable pass rates rather than explore an alternative learning and teaching approach – an effective solution in the current higher education environment that encourages the academic to prioritise other areas, such as research.

CHALLENGE #7: The context within universities is not favourable to exploration of innovative, alternative pedagogies.

Reflective journals – the Word document approach

During 2005, in an attempt to encourage student reflection and hopefully encourage a higher level of learning an assignment was introduced that required students to keep a “reflective” diary. The diary was submitted at the end of term for review by staff. There were two problems observed with this approach: creative writing and plagiarism. Since the diary was not submitted until the end of term, it was felt by teaching staff that many students left the diary until a last minute exercise in creative writing. There was also problems with plagiarism. The second offering of COIS20025 in 2005 had 62 plagiarism incidents.

To address these sorts of problems students were meant to submit their diaries three times during the term to teaching staff for checking. This practice was never widely implemented due to workload and resource problems. In particular, this is a problem for teaching staff at the international campuses who are either casual staff employed for a specific number of students and a particular set of assumptions, or they are full-time staff who operate under a high workload.

CHALLENGE #8: Workload pressures limit the adoption of changes in pedagogy.

The problem

Introduce the problem with the reflective journal assignment.

Back in 2006 I had a problem. I was about to take over a post-graduate Masters course in System development. The course was taught across multiple campuses by 10+ teaching staff and taken by 200+ students. Most of the students were from a non-English speaking background. In order to encourage reflection and higher order learning, the course had one assignment that required the students to keep a reflective journal. Here’s where the problem arises.

Students were told to keep the journal as a Word document and submit it at the end of the term. Consequently, most students did not work on the journal during term, leaving it to the last minute. Teaching staff did not see what was written in the journal and could not respond or comment. The course coordinator – located at a campus some distance from where the majority of the students were located – certainly couldn’t see what was going on. The limited integration between what the student was doing and the marking of the journal probably contributed to significant levels of plagiarism that occurred with the assignment.

How to make it better

Given my interest and experience with blogs, this seemed a perfect fit. In addition, at this stage this course was offered using CQU’s home-grown LMS, Webfuse. The Webfuse assignment submission system was already heavily used in this course, it was part of the way things were done. In addition, I am the main designer of Webfuse and as late of 2004 was the Webfuse team lead. This mean I could see how easy it would be to add a feature to Webfuse that would allow the use of blogs in a large course to be managed in a more efficient way. This feature was to be known as Blog Aggregation Management – BAM.

CHALLENGE #9 It was only through a unique combination of contexts that the idea for BAM was identified. Remove any one of these conditions and BAM probably would not have happened.

After 10 years of working on innovative e-learning through my own interest I was not going to develop BAM on top of the rest of my workload. Especially since there remained large uncertainty about the future of Webfuse (Webfuse eventually was repalced at CQU in 2010). In addition, at this stage my current laptop was getting on a bit. I needed a new one and university computer replacement policy would result in me getting something less than nice. I needed some money to top this up.

The previous coordinator of COIS20025 and I collaborated on a proposal to develop BAM. This was put to the faculty management and included the suggestion that I be given $2000 to develop BAM. The proposal was accepted.

CHALLENGE #10 The development of innovative blended learning pedagogy requires additional funding. Innovation costs initially.

The decision to support the development of BAM was, from my perspective, not a logical thing for the organisation to do. I believe it only happened because a small number of the players within faculty management at this point in time, had positive impressions of Webfuse, myself and the idea of BAM. Had this proposal been considered at another point in time, with different people in those positions, I believe the outcome would have been different.

The initial development and use of BAM

By the 8th of May 2006 I was running a “design workshop” for BAM. An hour long presentation to university staff proposing the idea and describing prototypes of how it might work. The aim was to get the idea out there and get feedback. The video of that presentation can be viewed on Google Video. I need to look through this and the powerpoint slides and find where I was up to with my thinking

On the 11th of September, 2006 a follow up presentation was given to report on the lessons learned so far in the use of BAM in the course. It includes a summary of what has happened as of Week 8 of (a 12 week) term:

  • 258 of 278 students have registered blogs;
  • Marking of blog posts not done, mostly due to the late arrival of the marking facility within BAM.
  • Staff overview of the blogs has been patch.

A significant component of the use of BAM in COIS20025 for this term was spent in designing the questions students were asked to respond to on their blogs. It wasn’t just the technology. The design had moved beyond just using minute papers into some specific questions intended to make visible the students’ preparation for the assignments. The aim was to enable staff to make comments before assignment submission, reduce plagiarism and attempt to increase higher-order learning.

Problems

For various reasons, I do not believe that these aims were achieved on a broad scale. In part, this is because the COIS20025 students were very pragmatic, the majority did what what was required to get the mark they wanted with a minimum of effort, rather engage completely with the intent.

CHALLENGE #11: A significant percentage of students are very pragmatic. They don’t automatically engage with the aims of innovative pedagogy.

Another significant contributor was “change management” issues. Essentially, most staff did not look at student blogs during the term. They were left until marking was done at the end of term. Two main issues created this: the late finish of the interface staff would use to mark the blogs, and workload issues for the staff. While BAM may have reduced the actual work required to mark student journals, the novelty and uncertainty around the system meant this was not how staff perceived the system.

CHALLENGE #12: New or novel pedagogies are always perceived to require more work from participants. This is simply because of the need for participants to become familiar with the novelty. Novelty on top of an overloaded or overly pragmatic set of participants limits engagement.

Successes

One of the more engaged members of the teaching staff made this observation in interviews

Students tend to fall off the wagon…the major reason why the blog was created. So that we can keep track of students working week by week rather than…working the night before the assignment is due…in that respect it seems to have worked well.

The blogs also appeared to confirm Du and Wagner (2005) and provide a useful predictor for learning outcomes for high and low performing students. Of the 9 high performing students in COIS20025, all 9 performed highly on their blog posts. Of the 6 students who did not register a blog (obvious by week 4 of term), 4 failed every course they studied that term and 2 failed two of the three courses they were studying.

Technically the system worked well. Students had a blog without the institution needing to provide a blog service. The management system, once in place, enable marking and tracking of student posts to occur and integrated with institutional management systems. The project was included in the ELI Guide to Blogging (Coghlan, Crawford et al. 2007) released in 2007 which indicated that the marriage of Web 2.0 applications with institutional systems was a compelling part of this project and that it promised to give institutional systems greater efficacy and agility with reduced cost.

More details

More detail about the initial use of BAM in COIS20025 can be found in two publications. One, written in 2006 just after completion, and another written in late 2008 after a period of some reflection.

The story of BIM – Origins – blogs and minute papers

The following is a start of trying to capture the story of BAM and BIM. The aim is to use this as part of a case study for a paper for MoodleMoot’AU 2010.

As well as trying to re-create the story of BIM from various publications, email archives and presentations the aim of the following is also to identify the challenges faced while developing BIM. This will be the first in a sequence of posts as I attempt to re-create the history.

Origins: Blogs and minute papers

I’d first used blogs in teaching around 2004. Had been aware of them for a while. By the end of 2005 I was back teaching first year programming in courses that were taken by distance education students, traditional on-campus students and international students at capital city, commercial campuses. I was directly responsible for the on-campus and distance education students.

For a long time (going back to the early 90s) I’ve been concerned with the plight of distance education students and the limited support they receive. In an attempt to achieve a couple of aims I wrote a paper titled “Enhancing the learning journey for distance education students in an introductory programming course”. It was intended to try and reinvigorate the REACT process while at the same time giving me the reason to reflect on how I might support the distance education students better and reduce the failure rate in the first year programming courses.

A part of the considerations was the use of blogs. Especially in combination with the idea of minute papers (Murphy & Wolff, 2005). The idea was described as this (Jones, 2005)

The author has used minute papers in face-to-face teaching and found them to be useful. It is thought that asking distance education students to blog a minute paper each time they do some study will provide a minimal level of structure and help the coordinator be aware of how each student is progressing.

Implementation plan

I implemented some of the ideas, including the use of blogs, described in Jones (2005) in the first term of 2006 in a course called Procedural Programming. Before the start of term I emailed all of the distance education students taking the course explaining how and why I wanted to use the blogs to support them. The reward to students for blogging minute papers was that I would be responding directly to their quotes. This included the offer of a pre-submission check of their assignments. i.e. before they submitted, I would look at their assignment and give them feedback.

There were only 17 or so distance education students, so while it was more work, I didn’t think it would be that great. However, I did recognise that it would be additional work, I thought it worthwhile as it would help the students, but there was no way to get recognition for this extra workload. Even though the rationale for doing was sound.

CHALLENGE #1: Organisational processes and policies, especially workload, don’t easily respond to changes in workload balance.

Checking my email archives for the course, I find the first mention of this blog. Apparently, I set up this blog to serve as an example for the students in this course. Here’s the first post reflecting on the early stages of this process.

Those same email archives reveal 3 students sending me the URLs for their blogs.
1 of those blogs is empty, no posts. Another has a couple of posts from the student and some comments from me. I remember this student. We ended up moving discussions to email and those discussions were quite extensive. The last blog had a regular post each week, for most weeks, and also had a comment from me on each one. Is that a 33% success rate? Of the students one failed (no posts), one got a credit (2 posts) and one got a HD (lots of posts). Is that a correlation between engagement on the blog and final result? It’s been since suggested that blogs are a good predictor of student results for high and low achievers.

Another student at this stage commented in an email

I dont know anyone who uses Blogs, and i have never read or used one myself.

CHALLENGE #2 Blogs are novel to many students. Students are conservative, getting them to try something new is a challenge.

Other comments from the same student

If you were offering me presubmission checks in return for a weekly minute sheet/blog i would do it. I can see how a student that was lacking confidence would feel more secure posting a message that only the lecturer would read. The only problem that i see with not having a communal learning environment is that if the lecturer cant perform their duties in a timely manner there is no one else that is aware of the students problems. The onus is fairly heavily on the lecturer to perform.

CHALLENGE #3 A design that relies on the academic to perform, will create a workload and a single point of failure.
CHALLENGE #4 Students are pragmatic. There needs to be reason for them to do something, especially something new.
CHALLENGE #5 The open nature of blogs can create some fear, especially for those who are less confident (this may/could be linked to what is known about shy students posting to discussion forums i.e. they tend not to)

The same student commented

The thing that i found that reduced the “transactional distance” was simply having an email/post/message from someone that called me by my name. The feeling that someone knows your name rather than your student number made a more profound difference to my study experience than i would have thought likely.

One of the points to remember is that under this model the distance education students were receiving a different type of support than the on-campus students.

At the moment, it appears that CQU’s course evaluation system has no record of any evaluations for this course. Will have to dig into that.

Conclusions

On the whole, this use of blogs was not a great success in terms of overall numbers. However, it did provide an opportunity to experiment, to test the waters. Obviously this experiment was largely positive because of what came next.

References

Jones, D. (2005). “Enhancing the learning journey for distance education students in an introductory programming course.” from https://davidtjones.wordpress.com/?attachment_id=496.

Murphy, L. and D. Wolff (2005). “Take a minute to complete the loop: using electronic Classroom Assessment Techniques in computer science labs.” Journal of Computing Sciences in Colleges 21(1): 150-159.

Improving L&T at Universities – The emperor has no clothes

The following comes from a place of frustration. The approaches universities are using to improve learning and teaching don’t work. But they continue to be held in reverence because they have become a purpose proxy. The people within universities charged with improving learning and teaching are no longer focused on or measured by improving learning and teaching. They are focused on and measured by the purpose proxy. i.e. how many L&T seminars they have run, how many teaching awards they’ve given out etc.

It wouldn’t be so bad if there weren’t already fairly significant amounts of research to show that they don’t work.

Common methods

The most common methods adopted with universities to improve learning and teaching (that I’ve seen) are:

  • Require staff (but only new staff) to get formal learning and teaching qualifications.
  • Run special L&T seminars.
  • Hand out L&T grants.
  • Give out teaching awards.
  • Develop new policies, standards etc and expect academics to meet them.
  • Communities of practice.

I don’t think any of them work, in terms of making a noticeable widespread (i.e. approaching 50%) improvement in L&T.

Evidence

Let’s look at teaching awards. Essentially giving rewards to the individual teachers for their quality.

Being good corporate citizens we in universities are applying what the business world has found. According to this post quoting from this book there have been three articles in the Harvard Business Review on incentive pay and organisational performance, the common point between all three

compensating people for only individual performance creates more problems than it solves, so rewards should emphasize organizational, not just individual, performance

So, while teaching awards probably aren’t strictly incentive pay, they are the sectors attempt to provide some incentive to be a good teacher and they do suffer the same problem.

To make it worse, they don’t work. I could quote on and on from studies of higher education which show that academics still perceive research to be the most rewarded activity. The perception is that teaching is not rewarded. And lots not forget the rewards for research are essentially individual, the good researcher gets promoted. Rewards for individual performance anyone?

As for the other strategies?

  • Formal qualifications.
    If formal teaching qualifications == good teaching, then the education programs at a university should be bursting at the seams with consistent, good quality, innovative teaching. Ahh, no.

    While it is important to improve/change the conception of learning/teaching of the teacher in order for change to happen (Ho, Watkins et al. 2001). There is little evidence to show that pedagogue’s conceptions of teaching will develop with increasing teaching experience or from formal training (Richardson 2005). Pedagogue’s approaches to teaching change slowly, with some change coming after a sustained training process (Postareff, Lindblom-Ylanne et al. 1997). While pedagogue’s are likely to adopt teaching approaches that are consistent with their conceptions of teaching there may be differences between espoused theories and theories in use (Leveson 2004). While pedagogues may hold higher-level view of teaching other contextual factors may prevent use of those conceptions (Leveson 2004). Environmental, institutional, or other issues may impel pedagogues to teach in a way that is against their preferred approach (Samuelowicz and Bain 2001). While conceptions of teaching influence approaches to teaching, other factors such as institutional influence and the nature of students, curriculum and discipline may also influence teaching approaches (Kember and Kwan 2000). Prosser and Trigwell (1997) found that pedagogue’s with a student-focused approach were more likely to report that their departments valued teaching, that their class sizes were not too large, and that they had control over what was taught and how it was taught. Other contextual factors that frustrate pedagogues’ intended approaches to teaching may include senior staff with traditional teacher-focused conceptions raising issues about standards and curriculum coverage and students who induce teachers to adopt a more didactic approach (Richardson 2005).

  • L&T seminars and L&T grants.
    I haven’t summarised the literature yet, but the little I’ve read backs up my experience. The people that apply and attend L&T grants and seminars are the same people. They are also, generally, the people who don’t need these. They are already doing good work. They should be supported but these activities are never going to reach the vast majority of academics.
  • New practices, policies etc.
    While academics see the greatest rewards from research, they will game the system. Policies won’t change that.
  • Communities of practice.
    I haven’t seen research on this, but I’ve seen what’s happened in the local context (so the foundation of this particular argument is fairly weak). Initially the CoP here were based around a particular method or interest (portfolios, active learning, peer learning etc.). That is examples of Technology I or II and technological gravity. None of these had any demonstrable widespread (approaching 50%) difference.

    The CoP approach has now morped to a focus on heads of program. Ad hoc feedback indicates a much greater engagement. Which is not surprising given that the head of program job is a thankless task and that the incumbents in these positions have a set of common problems. There’ll probably be some good work done here.

    However, it will be limited in its reach – mostly the heads of program – and will face the same problem. You have to get individual academics to change their practice. The heads of program may be in a better position than most, but I think they’ve still got a snowball’s chance if the context is not conducive to change. In addition, this CoP runs the problem of enshrining an “us and them” mentality within the heads of program. Get a group of people with similar problems together and they will share lots of stories about those problems. This will get the group feeling like one. However, the more stories they tell will lead them towards negative thoughts about the others – the non-head of program, teaching academic. This will start to colour their actions and thoughts. In much the same way it’s already coloured other folk in management roles.

The alternative

People’s beliefs and actions about what is right, what is accepted practice arises from their day to day experience. The day to day experience of academics within the currently learning context at universities teaches them what is right and what is accepted. It’s not focusing on teaching and learning.

If you change their day to day experience, you can change what is right and what is accepted. Guskey (1986; 2002) suggests it is the experience of successful implementation that changes the attitudes and beliefs of pedagogues. Pedagogues believe change will work because they have seen it work and this experience is what changes their conceptions of teaching and learning (Guskey 2002).

This isn’t about big bang changes, because they rarely work, are rarely successful and consequently rarely change conceptions. Instead it is about a system that focuses on the day to day experience of the academic and helps make it better. That shows that this day to day experience is important to the university and consequently should be important to the academic. A continual experience of successful change that makes their life easier and improves the quality of teaching and learning in ways that are consistent with teachers conceptions lays the groundwork for on-going change in conceptions.

If universities continue to focus on the wrong time frame, I believe they will continue to fail to make improvements in learning and teaching.

Some of the above strategies may be useful, but only after the system has started to successfully focus on the day to day experience of the academic. They’ll never be truly useful (in terms of gaining levels of approvment approaching 50% of all teaching) before that focus occurs. Before the focus occurs, the above approaches may well help a small percentage of staff, but they ignore the vast majority. And I feel that this is wrong. Especially when the small percentage success is trumpeted as major achievement while hiding the reality of very, very limited overall change.

References

Ho, A., D. Watkins, et al. (2001). "The conceptual change approach to improving teaching and learning: An evaluation of a Hong Kong staff development programme." Higher Education 42(2): 143-169.

Kember, D. and K.-P. Kwan (2000). "Lecturers’ approaches to teaching and their relationship to conceptions of good teaching." Instructional Science 28(5): 469-490.

Leveson, L. (2004). "Encouraging better learning through better teaching: a study of approaches to teaching in accounting." Accounting Education 13(4): 529-549.

Postareff, L., S. Lindblom-Ylanne, et al. (1997). "The effect of pedagogical training on teaching in higher education." Teaching and Teacher Education 23(5): 556-571.

Prosser, M. and K. Trigwell (1997). "Relations between perceptions of the teaching environment and approaches to teaching." British Journal of Educational Psychology 67(1): 25-35.

Richardson, J. (2005). "Students’ approaches to learning and teachers’ approaches to teaching in higher education." Educational Psychology 25(6): 673-680.

Samuelowicz, K. and J. Bain (2001). "Revisiting academics’ beliefs about teaching and learning." Higher Education 41(3): 299-325.

The value of lurking – another research idea for the indicators project

The question of the value of lurking is a common one for learning and teaching. Is it okay for students to simply sit quietly and listen, especially in an online forum? Should we “force” them to participate?

I’m currently reading Saba (2000) as a ploy to generate more insight into “what is research”. Saba (2000) has a section titled “Theory-based research”, which summarises some of the research done in distance education. The first cab off the rank is Fulford and Zhang (1993) and this research seems to suggest one answer to the question of lurking.

Learners’ perception of learning

Saba’s (2000) description of their research is that it studied the perception learners’ had of interaction. The conclusion was that such perception was a critical indicator of satisfaction. Saba quotes

overall interaction dynamics may have a stronger impact on learners’ satisfaction than strictly personal participation. Vicarious interaction may result in greater learner satisfaction than would the divided attention necessary to ensure the overt engagement of each participant

The recommendation then is to design teaching so as to increase the learners perception of overall interaction.

That sounds like to me as offering an answer to the lurking question. There is some benefit to lurking if through lurking the learner perception of the level of interaction increases.

The indicators project

The indicators project seeks to mine the usage data of LMSes to identify useful patterns. SNAPP is a project looking at generating “maps” of student interaction within discussion forums.

Combining these with some other methods would seem to be a good way to test these findings. Especially given the evolution of distance education since 1993.

Should probably start with the other literature. According to Google Scholar Fulford and Zhang have been referenced almost 300 times. Should be some interesting stuff in there.

Beaudoin (2002) might be a good place to start, from the abstract

The data shows that these students do, in fact, spend a significant amount of time in learning related tasks, including logging on, even when not visibly participating, and they feel they are still learning and benefiting from this low profile approach to their online studies. However, preliminary analyses of course grades indicate that the mean grade is slightly better for high visibility learners than for no visibility learners. Findings suggest that further research in the area of the so-called invisible learner is a critical area of investigation to better understand the dynamics of asynchronous learning and teaching.

Looking at the invisible learner might be interesting.

References

Beaudoin, M. (2002). “Learning or lurking?: Tracking the “invisible” online student.” The Internet and Higher Education 5(2): 147-155.

Fulford, C. P., & Zhang, S. (1993). Perception of interaction: The critical predictor in distance education. The American Journal of Distance Education, 7(3), 8-21.

Saba, F. (2000). “Research in distance education: A status report.” International Review of Research in Open and Distance Learning 1(1).

Different perspectives on the purpose of the LMS

Antonio Vantaggiato gives one response to a post from Donald Clark titled “Moodle: e-learning’s frankenstein”. Clark’s post is getting a bit of traction because it is being seen as a negative critique of Moodle.

I think part of this problem is the failure to recognise the importance of the perceived purpose to which Moodle (or any LMS) is meant to serve. Just in my local institution, I can see a number of very different perceptions of the purpose behind the adoption of Moodle.

In the following I’m stealing bits of writing I’ve done for the thesis, some of it has appeared in previous posts. This probably makes the following sound somewhat pretentious, but I’ve gotta got some use out of the &%*#$ thesis.

The importance of purpose

Historically and increasingly, at least in my experience, the implementation of e-learning within universities has been done somewhat uncritical with the information technology taken for granted and assumed to be unproblematic. This is somewhat surprising given the nature of the universities and the role academics are meant to take. However, in my experience the selection of institutional LMSs is driven by IT and management with little room for critical thought or theory informed decision making.

Instead they rely on a very techno-rational approach that takes a very narrow perspective of what technology is, how it has effects and how and why it is implicated in social change (Orlikowski and Iacono 2001). A different perspective is that technology serves the goals of those who guide its design and use (Lian 2000).

This is important because many, if not most, universities follow, or at least profess to follow, a purpose driven approach to setting strategic directions (McConachie, Danaher et al. 2005). The implementation of an LMS is being done to achieve specific institutional purposes. The very definition of a teleological design process is to set and achieve objectives, to be purpose driven (Introna, 1996). When an institution engages in selecting an LMS, the purpose is typically set by a small group, usually organisational leaders, who draw on expert knowledge to perform a diagnosis of the current situation in order to identify some ideal future state and how to get there.

Once that purpose is decided, everything the organisation does from then on is about achieving that purpose with maximum efficiency. By definition, any activity or idea that does not move the organisation closer to achieving its stated purpose is seen as inefficient (Jones and Muldoon, 2007).

Differences of purpose

Many of the folk responding to Clark’s post who are defending Moodle have their own notion of the purpose of Moodle, usually how they have used it. Others draw on the purposes espoused by the designer(s) of Moodle. There is little recognition that there exists a diversity of opinions about the purposes of Moodle.

A little of this diversity is represented in discussions about how Moodle is used in individual courses. For example, this comment mentions that Moodle does teacher centered very well. i.e. if a teacher sees the purpose of a course site to distribute information, Moodle can do that. This comment makes the point that Moodle is a tool, the pedagogy is not about the tool, it is about the approach.

Now, while to some extent that is true, I also agree with Kallinikos (2004) that systems can have profound effects on the structuring of work and the forms of human action they enable or constrain.

While Moodle’s designers may have all sorts of wonderful intents with the purpose of Moodle. Within a university the purpose assigned to Moodle by the people implementing it and supporting play a significant part. The processes, structures etc that they put around Moodle within an institutional setting can enable or constrain the purpose seen by the Moodle designers and the purpose seen by the staff and students who will use it.

Moodle/LMS as an integrated enterprise system

Due to the complexity of implementing Moodle for a largish organisation the people driving Moodle implementations within universities are usually IT folk. It is my suggestion that the purpose they perceive of Moodle is that of an integrated, enterprise system. A university’s LMS forms the academic system equivalent of enterprise resource planning (ERP) systems in terms of pedagogical impact and institutional resource consumption (Morgan 2003).

To slightly paraphrase Siemens (2006), the purpose of an LMS for the institution is to provide the organisation with the ability to produce and disseminate information by centralising and controlling services. The LMS model with its nature as an integrated, enterprise system fits the long-term culture of institutional information technology and its primary concern with centralizing and controlling information technology services with a view to reducing costs (Beer and Jones 2008).

An LMS is designed to provide an organisation with all the tools it will need for e-learning. Weller, Pegler and Mason (2005) identify two approaches to the design of an LMS: monolithic or integrated approach, and the best-of-breed approach. The monolithic approach is the predominant approach and seeks to provide all common online learning tools in a single off-the-shelf package (Weller, Pegler et al. 2005).

The evidence of this purpose can be seen when you go to your LMS folk and ask them “I’d like to do X”. The response to this question will generally be not what is the best marriage of pedagogy and technology (the best blended learning) to achieve your goal. The response to this question will generally be “How to do X in the LMS”. Regardless of how much extra work, complexity and just plain inappropriateness doing X in the LMS requires.

If all you have is an LMS, every pedagogical problem is solved by the application of the LMS.

What should the purpose of an LMS be?

BIM is a representation of what I think the purpose of an LMS should be. i.e. the LMS should provide the services that are necessary/fundamental to the university/institution, and only those. Increasingly, most of the services should be fulfilled by services and resources that students and staff already use and control.

BIM provides academics teaching a course a way to aggregate blog posts from students and, if they want to, mark them. Those marks are integrated into the Moodle gradebook. The assumption is that marking/accreditation is one of the main tasks a university performs and that there aren’t external services that currently provide that service.

There are, however, a great many very good and free blog services. So students use their choice of blog provider (or something else that generates RSS/Atom) to create and manage their contributions.

The purpose of the LMS isn’t to provide all services, just those that are required for the institution’s tasks.

Eventually, the term LMS becomes a misnomer. The system isn’t about managing learning. It’s about providing the glue between what the institution has to provide and what the learners are using. The purpose is about achieving the best mix of pedagogy and technology, rather than on how to use the LMS.

This perspective obviously has connections with Jon Mott’s (2010) article and the various folk who have written about this previously.

References

Beer, C. and D. Jones (2008). Learning networks: harnessing the power of online communities for discipline and lifelong learning. Lifelong Learning: reflecting on successes and framing futures. Keynote and refereed papers from the 5th International Lifelong Learning Conference, Rockhampton, Central Queensland University Press.

Jones, D. and N. Muldoon (2007). The teleological reason why ICTs limit choice for university learners and learning. ICT: Providing choices for learners and learning. Proceedings ASCILITE Singapore 2007, Singapore.

Kallinikos, J. (2004). “Deconstructing information packages: Organizational and behavioural implications of ERP systems.” Information Technology & People 17(1): 8-30.

Introna, L. (1996). “Notes on ateleological information systems development.” Information Technology & People 9(4): 20-39.

Lian, A. (2000). “Knowledge transfer and technology in education: Toward a complete learning environment.” Educational Technology & Society 3(3): 13-26.

McConachie, J., P. Danaher, et al. (2005). “Central Queensland University’s Course Management Systems: Accelerator or brake in engaging change?” International Review of Research in Open and Distance Learning 6(1).

Morgan, G. (2003). Faculty use of course management systems, Educause Centre for Applied Research: 97.

Mott, J. (2010). “Envisioning the Post-LMS Era: The Open Learning Network.” EDUCAUSE Quarterly 33(1).

Orlikowski, W. and C. S. Iacono (2001). “Research commentary: desperately seeking the IT in IT research a call to theorizing the IT artifact.” Information Systems Research 12(2): 121-134.

Siemens, G. (2006). “Learning or Management System? A Review of Learning Management System Reviews.” from http://ltc.umanitoba.ca/wordpress/wpcontent/uploads/2006/10/learning-ormanagement-system-with-reference-list.doc.

Weller, M., C. Pegler, et al. (2005). “Students’ experience of component versus integrated virtual learning environments.” Journal of Computer Assisted Learning 21(4): 253-259.

Focusing on the wrong time frame – a core problem of university L&T?

I think that the quality of most university learning and teaching is somewhat less than it could be. There are some really good examples and really engaged teaching academics. However, they are in the minority. This problem is made worse when the context of a particular university is extremely complex. For example, a number of Australian public universities, in response to reductions in government funding, have developed a collection of partnerships with commercial and overseas organisations to offer their courses in various modes. Some of these universities also have traditional on-campus students as well as distance education student spread throughout the globe.

My experience at one such institution suggests that the focus on improving learning and teaching is being focused on the wrong time frame. The decision makers and organisational units that support teaching and learning are don’t focus on the truly long term (5/10+ years – too long) or the truly short term (every day – too short). Their focus is on a period ranging from a single term/semester (who’s teaching what, what results did students get) through to about 2 or 3 years (creation of policy and implementation of big bang improvement projects). This period is seen to be just right.

For example, after a long period of discussion a new policy will be promulgated. But, little or no effort will be made to implement the policy in a way that can be operationalised. Little or nothing will be done to understand the impact on the day to day life of students and staff.

The problem

As a result there is no focus on what academics do day to day. On what they experience, the problems the face or the stories they tell each other to create reality. Instead policy is informed by the experience of those at the term to 3 year time frame.

Policy and processes are not being set based on knowledge of the day to day, but on experience divorced from the day to day. It doesn’t engage with the reality of the teaching and learning context. It doesn’t connect with what is important to the students and staff living that day to day experience.

That disconnect means that:

  1. It is based on a lack of knowledge, a lack of appreciation. i.e. the decisions are faulty.
  2. The disconnect is obvious to the day-to-day people and it discourages them. Not a good basis on which to build behaviour change (which is what improving learning and teaching requires).

One solution

Engaging with the problems of the day to day experience is what Webfuse did at its best.

It’s an approach that focused on improving the quantity and quality of the connections around the day to day experience of teaching and learning. And this is what learning is all about.

Webfuse is dead! Long live Webfuse!

Since about 1996 much of my research and development around e-learning has found “concrete” form in a system call Webfuse. First described in Jones and Buchanan (1996) Webfuse became an essential part of CQU’s operations. The unkindest (and basically wrong) description of Webfuse is that it was a proprietary, single institution learning management system.

Based on the definition, Webfuse is dead.

CQU is adopting Moodle as its LMS and folk are actively seeking to remove all the other Webfuse services that people use.

Why it worked

What most folk don’t get is that the important part of Webfuse was never the system. It was the process. It was having a small group of people employed to help the academics. It also helped that the group had, at least for a time, little or no direction from senior management who though they knew what to do. This allowed the group helping academics to respond to the academics context, to bring in changes and systems that made life easier for the academics.

This was possible because the group understood the technology and were able to put in place lots of rapid, but small, changes to the product in response to the needs of the staff (and students). This meant that academics could see the technology changing to make their life easier and allowing them to do new things.

This rapid change was only possible because there were no barriers between the developers – the people who could change the technology – and the staff and students – the people using the technology. The developers saw first hand in conversations and in the tea room what the academics were going through. They had deep knowledge of the context.

The developers had a much greater commonality of meaning with the academics than traditionally possible.

Just one simple example of the benefits this bring can be found in the Webfuse online assignment submission system (Jones and Behrens, 2003). The features of this system fit the local contextual needs and offer a different set of features than in any other LMS-based online assignment submission system. It’s not a copy, it’s different and it’s better. For example, CQU”s IT folk are currently modifying the Moodle assignment submission system to provide the same features.

Webfuse and its developers were not limited to course sites. Webfuse also contains an array of management/adminstrative type features that made life easier. They could make connections between different aspects.

Why it’s still needed

The traditional approach to e-learning within universities does the exact opposite of what was most important about Webfuse. Some of the differences include:

  • There are significant barriers preventing direct communication between a developer and a user leading to a huge difference in meaning, not commonality.
    A user never speaks directly to a developer. Most organisations don’t have a developer, if they do any communication must go through a few committees and layers within the IT division.
  • People helping academics can’t change the system.
    As a result of the first, when an academic has a problem the system can’t be changed. At best the person helping the academic can show them the roundabout, inefficient kludge that is possible operating within the confines of the system.

    Think about what this does to the relationship between the academics and the people helping them. It makes it very hard to seem helpful and build up a relationship of trust.

  • It destroys innovation.
    Since the institution can’t change the system in response to contextual needs, they can’t innovate. They are stuck following along behind everyone else. They can’t innovate until someone else has developed, tested and proven that the tool works.
  • The focus on course delivery creates disparate systems.
    Moodle is only being used to deliver courses. The course sites generally only exist for the term. Other administrative tasks like creating course profiles/outlines, curriculum mapping etc. are seen as separate tasks looked after by separate parts of the IT organisation.

    This contributes to the problem you see at most universities. The LMS used for a small part and then a plethora of other systems for administrative tasks which have to be – and usually aren’t – integrated with the LMS.

  • Top-down, disconnects as the basis of policy.
    In the absence of bottom up drive any and all change within the institution gets driven from outside (responding to government) and from the top down. From people and places that don’t know the complexities and details of the context of academics. Hence their plans are flawed, they are based on too little knowledge.

Long live Webfuse

There is still a need for people who work very closely with the academics, can appreciate their context, can establish a commonality of meaning about what their experiencing and have the knowledge and capability to modify the technology within the system to improve that context, to innovate and try new things. At least if an institution truly wants to improve the quality of its learning and teaching.

Long live Webfuse!

References

David Jones, Sandy Behrens, “Online Assignment Management: An Evolutionary Tale,” Hawaii International Conference on System Sciences, vol. 5, no. 5, pp. 156c, 36th Annual Hawaii International Conference on System Sciences (HICSS’03) – Track 5, 2003.

David Jones, Renay Buchanan, The Design of an Integrated Online Learning Environment, Making New Connections, Proceedings of ASCILITE’96 Adelaide, Allan Christie, Patrick James, Beverley Vaughan, pp 331-345