Category Archives: curriculumMapping

Strategies for curriculum mapping and data collection for assuring learning

The following is a summary of and some reaction to the final report of an OLT funded project titled “Hunters and gatherers: strategies for curriculum mapping and data collection for assuring learning”. This appears to be the project website.

My interest arises from the “Outcomes and Analytics Project” that’s on my list of tasks. That project is taking a look at Moodle’s new outcomes support (and perhaps other tools) that can be leveraged by a Bachelor of Education and try to figure out what might be required to gain some benefit from those tools (and whether it’s worth it).

Reactions

The recommended strategies (holistic, integrated, collaborative, maintainable) could form a good set of principles for some of what I’m thinking.

In terms of gathering data on student performance, assessment and rubrics appear to be the main method. Wonder if analytics and other approaches can supplement this?

It would appear that no-one is doing this stuff very well. The best curriculum mapping tool is a spreadsheet!!! And the data gathering tools are essentially assignment marking tools. Neither set of tools were well evaluated in terms of ease of use.

Lots of good principles and guidelines for implementation, but crappy tools.

Student centered much?

AoL is defined as determining program learning outcomes and standards and then gathering evidence to measure student performance. use for curriculum development, continuous improvement and accrediatation – but no mention is made of helping students develop eportfolios for employers. Especially in professional programs with national standards, this would seem an obvious overlap. Student centered much?

Interesting given that AoL is meant to be based on student centered learning.

Staff engagement

Is positioned as a difficulty.

Facilitating

AN earlier project (on which this was built) was titled “Facilitating staff and student engagement with graduate attribute development”. Apparently limited to helping staff design criteria to assess GAs and students self-evaluate against those. Seems to be only a small part of the facilitation process.

GAs versus professional standards

I have to admit to being deeply skeptical around the notation of institutional graduate attributes. Always struck me as a myth created by high-priced senior management to justify what was unique about the vision they were creating and hence with little connection to the reality of teaching at a university. In particular, that the standards/attributes set by the professional bodies associated with certain disciplines would always count for more.

Of course institutional GAs have been apparently required by the government since 1992. I wonder if this is like some of those other legal requirements that have been discovered to have never existed, ceased to exist or which were misinterpreted?

Executive summary

Assurance of Learning (AoL) “evaluates how well an institution accomplishes the educational aims at the core of its activities”…AoL provides “qualitative and quantitative indicators for the assessment of the quality of award courses” Thus be used for

  • institutional/management ends: strategic directions, priorities, quality assurance, enhancement processes.
  • individual curriculum development.
  • Valid evidence to external constitutents.

Focus of this project on

  1. mapping program learning outcomes.
  2. Collecting data on student performance against each learning objective.

Aside: interesting that they’ve already used both outcome and objective. Does this mean these are different concepts, or just different labels for the same concept?

Investigation was done via

  • Exploratory interviews with 25 of 39 associate deans L&T from Oz Unis.
  • 8 focus groups with 4 good practice institutions, 2 at each institution – one with a senior leader, the other with teaching staff.
  • Delphi method – but who with?
  • Interviews with experts.
  • online survey.

Recommended good practice strategies

  • holistic – whole of program, to ensure student’s progress and introduction of GAs prior to demonstration.
  • integrated – GAs must be embedded in the curriculum and links to assessment.
  • collaborative – developed with teaching in an inclusive – not top down – approach to engage staff.
  • maintainable – must not be reliant on particular individuals or extra-ordinary resources.

They then proceed to mention typical cultural change strategies.

Also did an independent review of existing tools. Interestingly the Blackboard 9.1 goals/standards service gets a mention.

Chapter 1 – Project overview

Identifies 7 key stages in assuring learning from an AACSB White Paper

  1. establishing graduate attributes and measurable learning outcomes for the program;
  2. mapping learning outcomes to suitable units of study in the program (where possible allowing for introduction, further development and then assurance of the outcomes);
  3. aligning relevant assessment tasks to assure learning outcomes;
  4. communicating learning outcomes to students;
  5. collecting data to show student performance for each learning objective;
  6. reporting student performance in the learning outcomes;
  7. reviewing reports to identify areas for program development (‘Closing the Loop’).

Explains the growing requirement for this, lots of acronyms and literature.

Mentions their prior project which development the ReView online assessment system to help staff develop criteria that assessed GAs within the set assignments. Students have self-evaluations.

Project aims to inform strategy to identify efficient and manageable assurance mechanisms (effective not important?).

Chapter 2 – methodology

Key guiding questions

  1. How is mapping of GAs being done?
  2. How is the collection of GA data being done?
  3. What are the main challenges in mapping and collecting?
  4. Are there identifiable good practice principels?
  5. What are the tools currently being used?

Chapter 3 – Literature review

here come the standards

Standards are defined as “the explicit levels of attainment required of and achieved by students and graduates, individually and collectively, in defined areas of knowledge and skills” (TEQSA, 2011, p. 3)……Academic standards are learning outcomes described in terms of core discipline knowledge and core discipline-specific skills, and expressed as the minimum learning outcomes that a graduate of any given discipline (or program) must have achieved (Ewan, 2010).

TEQSA is apparently requiring academic standards “be expressed as measurable or assessable learning outcomes”.

Determining standards and then collecting data against those is complex. Coates (2010) acknowledges the complexity and suggests a need for cultural change. And there is apparently an urgent need for “new, efficient and effective ways of judging and warranting” (Oliver, 2011, p. 3).

Extant literature

AoL finds its pedagogical basis in student-centered learning.

Curriculum mapping in AOL is the process of embedding learning outcomes related to GAS into units of study where these are introduced, developed and then assured.

AUQA required curriculum mapping as do most professional accrediting bodies – hence the 2009 observation from Barrie (et al 2009) that most Australian universities have some sort of strategic project underway.

The higher ed mapping literature is scant but suggests it’s useful for (all backed up with citations)

  • identifying gaps in a program
  • monitoring course diversity and overlap
  • providing opportunity for reflection and discourse
  • reducing confusion and overlap and increasing coherence

There are more, but there does seem to be some overlap.

Mentions the problem of the compliance culture, others include

  • difference between the intended and the enacted curriculum from the students’ perspective
  • how to contextualise GAs into a discipline.
  • mapping seen as threatening, as a course cutting exercise, criticisms of teaching material etc.
  • labour intensive exericse.

Staff engagement is seen as the key and current suggestions for improvement include

  • develop a conceptual framework for developing GAs, including 3 elements
    1. clear statement of purpose for curriculum mapping.
    2. a tool that allows an aggregate view of a course.
    3. a process for use of the tool
  • map GAs using extensive audits of each course.
  • a cyclical process including visual representations to enable a fluid/adaptable curriculum
  • availability of sufficient resources.
  • use of alignment templates (isn’t this a tool?)
  • professional development to integrate and contextualise GAs.
  • having specialists who can teach a particular attribute.
  • whole of program approach, focus on team co-operation and more time spent on design.
  • staff support where workloads increase.
  • linkages between GA development and professional development.

Embedding versus standardised testing

Mention of various standardised tests at the end of study. Talks about plusses and minuses.

Data collection for AoL

Focused on entering student performance outcomes against each learning objective. Need a “systematic method to collect data and explore the achievement levels of students in each of the selected attributes” to inform on-going development.

There are challenges in collecting and providing evidence – highlighting the need for efficiency and streamlining the process.

Assessment rubrics (formative and summative) are key. But there are challenges. Don’t want a “tick list”. Some skills are ill-defined, overlapping and difficult to measure. And the question of standardisation – homegenisation or pursuit of common goals. Multiple interpretations of criteria.

Rubrics can become to be used for comparison between institutions; assurance of content/process/outcomes across courses.

Continuous improvement/closing the loop

Apparently the “raison d’etre for assessing student learning” and also something that institutions are “most confused about how to go about closing the loop (Martell, 2007)”…..”integration of the assessment of learning outcomes into developmental approaches in the classroom has been somewhat intangible (Tayloer et al, 2009)”.

Curriculum mapping

important features for selecting a CM system to support AoL

  • support an inclusive and participatory process;
  • foster a program-wide approach to produce a mapped overview;
  • map by assessment task;
  • develop student awareness of attributes and their distribution within the program;

The standout tool was a spreadsheet!

Data collection

Important features for a data collection system included

  • implement a consistent criteria for attributes across programs;
  • extract outcome-specific data;
  • embed measurement in the curriculum
  • produce built-in reports;
  • conduct analysis for closing the loop;
  • implement multiple measures of AoL for program wide view.

ReView was seen as the stand out. But then, it arose from the last OLT project. But then that makes this comment interesting

ReView does not rate that well on ‘ease of use without the need for much supplementary professional development

Integrating alignment into Moodle and academic practice: A proposal and a RFI

I’m off to the 2001 Australian MoodleMoot next week. The conference program includes a collection of 3 minute show and tell sessions on the Tuesday afternoon. The following is a summary of what I think I’m going to talk about and a call for suggestions.

I’m starting to add all the associated resources with the presentation to this post.

More information

Other resources/information around this idea include:

  • A blog post introducing how curriculum mapping might work in Moodle.
  • A detailed, draft grant proposal for a broader project around embedding mapping/alignment into a university.
    This proposal includes a fairly long reference list which points to some of the literature that informed this idea.

Video

The following video is a slightly extended version of the talk, using the same slides, recorded after the Moodlemoot.

Slides

The purpose

The title of this post is probably going to be the title of the talk. From that you can assume that this is not a show and tell of something that is working, but instead a proposal of an idea. The aim is to find out if there are other people interested in this project or already working on something similar. The aim is to start a conversation. The talk is also request for interest (an RFI). I’m keen to hear from folk interested in working on this idea, especially in terms of a potential ALTC grant for next year.

The proposal is based on previous ideas posted here. At the core is the idea of how curriculum mapping might work in Moodle. However, the intent is to do much more than simply modify Moodle. The broader aim is to modify the environment and processes within which teaching academics work in order that consideration of alignment (be it constructive, instructional, curriculum or graduate attributes) is part of every day practice.

A more detailed description of this idea is available here. The rest of this is a written summary of what I think the 3 minute show and tell will cover next week at the Moot.

The problem

Within Australian Universities, the alignment of what happens within a course (sometimes known as a unit) against some outcomes or graduate attributes is becoming widespread, even standard practice. For example, there’s a presentation at the Moot with the title “Translating Learning Outcomes in Moodle”. This presentation draws on Bigg’s (1996) idea of constructive alignment, which is probably the most common, currently used concept of alignment. The push toward graduate attributes for everything is perhaps the other common application of alignment within Australian higher ed.

The Moot presentation identifies as a problem the difficulty of translating learning outcomes into an effective course design within an LMS. The problem which I’m interested is connected to this, but is also a little different. The problem I’m interested in is that the every day, regularly experience of an academic doesn’t require them to think about alignment. More broadly, the everyday experience of teaching academics doesn’t encourage nor enable them to think about learning and teaching from an educational perspective. Instead the focus on low level tasks like uploading documents because of the low-level of abstraction in most LMS.

Experience is important

What people experience is important. There’s a growing body of literature from neuroscience (e.g. Zull, 2002) and psychology (e.g Bartunek and Moch, 1987) that suggests your experiences shape who you are, what you think and how you see the world. Which in turn is related to insights like Kolb’s learning cycle.

Kolb's Learning Cycle

If alignment is not something academics experience regularly, and experience within a context that encourages and enables them to reflect and experiment with alignment, then how are they expected really to learn and adopt alignment?

The proposal

The proposal aims to modify the environment in which academics operate such that they are encouraged and enabled to consider alignment as a regular component of their everyday teaching experience. To provide an environment in which they can move through all of the stages of Kolb’s learning cycle. The proposal is based on the following assumptions and propositions:

  • The most common teaching experience for university academics is teaching and slightly tweaking a course that has been taught before.
  • It is fairly simple to modify Moodle to enable the mapping of alignment relationships between Moodle activities and resources and outcomes or graduate attributes.
  • Once this alignment information is being maintained, an ecosystem of services can be added to Moodle that enable reflection, abstraction, and active testing of ideas around alignment in a collaborative and open way.
  • If such an ecosystem enabled and encouraged effective, on-going use, then the quality of learning and teaching would improve.
  • On-going use of such an ecosystem would raise interesting questions about the design and operation of Moodle.

Disclaimer: I have some reservations about alignment, however, it’s almost become a requirement within Australian higher education and I do believe that consideration of alignment could provide a useful McGuffin for learning and teaching.

The 3 minute show and tell will focus on showing some proposed screen shots of how curriculum mapping might work within Moodle and some initial ideas of how the resulting alignment information could be used to create an ecosystem of services.

Request for interest

Effectively implementing something like this is not easy. It would be improved by having a good combination of skills and perspectives. I’m keen to work with people who are interested in trying to further develop and eventually implement this idea.

I’m especially interested in hearing about projects that are related to, or already implementing something like this.

References

Bartunek, J., & Moch, M. (1987). First-order, second-order and third-order change and organization development interventions: A cognitive approach. The Journal of Applied Behavoral Science, 23(4), 483-500.

Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364.

Zull, J. (2002). The art of changing the brain. Stirling, Virginia: Stylus Publishing.

The alignment project as leadership

The following signals a slight change in direction around the curriculum mapping project. First, the project is now going under the label “alignment project” (curriculum mapping is just one aspect of the project). Second, the project is likely to be re-framed as an application for an ALTC leadership grant. This post is an attempt to begin this re-framing. It’s really just thinking out loud.

As a result, I am very interested in suggestions and criticisms. In terms of suggestions, I’m particularly keen for insights onto better/alternative theoretical frames. It has become a bit confused as I’ve tried out different lines of approach.

Note: In the following a “course” typically means the smallest unit of teaching offered by an institution. A “program” is a group of courses that form a degree or perhaps a major.

Summary of the alignment project

Alignment is an increasingly core component of teaching in Australian universities. At a basic level, alignment is where the learning resources, learning activities and assessment of learning all align with the stated outcomes or aims of teaching (in a course/unit or program/course). Such alignment is a core component of various “movements” within university learning and teaching including: graduate attributes, quality assurance, and improvements in learning and teaching. However, consideration of alignment is not a regular, everyday part of teaching or learning at universities.

The majority of academic teaching practice involves the teaching of an existing course, one the academic has usually taught before, and as such most “teaching practice” revolves around making minor modifications to material or content (Stark, 2000). Academics are not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988). Alignment is most considered during the development of new courses, major overhauls of existing courses or in response to external quality assurance needs.

The following two sub-sections try to summarise the alignment project. The first is a more concrete description, the second more abstract or theoretical. The thinking behind this project has changed many times, the following are likely to change. Suggest away.

Process and intent

As currently thought, the alignment project can be described as four main tasks:

  1. Modify Moodle to allow mapping of alignment.
    Moodle has been chosen because it is the institutional e-learning system at both institutions that are part of the project. As such, support for Moodle is embedded into the institution and will continue to be supported. In addition, Moodle is an increasing part of the everyday practice of teaching academics. Lastly, Moodle, for a number of reasons, is very flexible and easy to modify and any modifications made could be usable by other institutions. The point of this project is not the Moodle modifications, it’s simply the best solution for embedding these changes into the institutions.

    The changes will focus on enabling the alignment of outcomes (be the course learning outcomes, program learning outcomes, those from accrediting bodies, graduate attributes etc) with the assessment, activities and resources within a course site. Having this functionality is a foundation for the rest of the project.

  2. Work with teaching staff to map their courses.
    Mapping the alignment of a course within Moodle is not going to be straight forward. Teaching staff are likely to be busy and may not be entirely familiar with the concepts around alignment and mapping. The content and design of a Moodle course site may not be currently appropriate in terms of developing a useful mapping of alignment. Using the alignment mapping functionality added to Moodle may not be straight forward.

    For these and other reasons staff from the L&T support services will have to work collaboratively with academic staff to overcome these problems. This will be the first cycle of action research and will identify specific problems and insights into potential solutions.

  3. Work with teaching staff to embed alignment into everyday practice.
    Once the initial mapping of alignment is complete, the focus moves onto helping academics maintain and reflect on the level of alignment within their course and programs. On helping them embed alignment considerations into everyday practice.

    This is the second cycle of action research and will likely include the development of models, tools and processes that address questions such as:

    • How do you encourage reflection and action based on the everyday consideration of alignment?
      Identify the support, processes and tools do teaching academics and program leaders need to encourage and enable reflection and action?
    • How do you measure and give feedback upon action based on the everyday consideration of alignment?
  4. Develop on-going and embedded institutional processes that take the lessons learned from the everyday consideration of alignment and use it to remove barriers within the institutional context.

Theoretical perspective

The alignment project is seeking to take the first steps towards what Biggs (2001) called the reflective institution. It seeks to do this by modifying the institutional systems around learning and teaching in ways suggested by Biggs (2001). These are:

  1. Make obvious the quality model.
    Most institutions espouse the theory of alignment, however the teaching systems and processes employed by institutions do not make this theory explicit. The first step is to modify these systems so that a focus on alignment is made explicit as a part of every day teaching practice.

    This is partly achieved through the modifications to Moodle to enable mapping of alignment. But more importantly it is achieved through the changes in L&T support, systems and processes that support academic in using those new Moodle capabilities. In part, these changes are the next step.

  2. Provide appropriate support for quality enhancement.
    Further modify these systems and process to enable and encourage teaching staff to reflect and improve their teaching through a focus on alignment. The modifications build on the changes in the previous stage to enable this support to be highly contextualised to everyday teaching practice. A focus on helping in what academics do most often, the fine-tuning of existing courses.
  3. Inform the quality feasibility process.
    Quality feasibility is the removal of factors in the teaching environment that are not conducive to good L&T. The everyday consideration of alignment will identify a range of barriers in the institutional setting, many of which will require the engagement of institutional leadership to remove.

ALTC leadership grants

The guidelines for the ALTC leadership grants scheme describe the grants as being for (my emphasis added)

projects that build leadership capacity in ways consistent with the promotion and enhancement of learning and teaching in contemporary higher education, and which reflect the ALTC’s values of excellence, inclusiveness, diversity and collaboration, and its commitment to long-term, systemic change.

The scheme has three priorities which can be summarised as being focused on: institutional leadership capacity building; disciplinary and cross-disciplinary leadership; and building on earlier projects. The alignment project seems to be best suited to Priority one

institutional leadership to enhance learning and teaching through leadership capacity-building at the institutional level.
  • Funding range: $150,000 to $220,000
  • Project duration: up to 2 years

In completing a lit review around leadership for the ALTC, Southwell and Morgan (2009) make the observation that

Leadership for Excellence in Learning and Teaching Program is expected to be a ‘demonstrable enhancement of learning and teaching through leadership capacity building’

What is leadership?

It seems required when discussing leadership to make the observation that “leadership eludes comprehensive definition” (Southwell and Morgan, 2009). Southwell and Morgan (2009) reference Marshall (2006) and Jameson (2006) as folk who have made that observation. Having made this claim, the idea is that you then define your familiarity with the broad array of perspectives, understandings and definitions of what leadership is. I’ll postpone that bit for now.

Southwell and Morgan (2009) draw on Leithwood and Levin (2005) who suggest that the core of most conceptions of leadership are two functions generally considered to be indispensable:

  1. Direction-setting: helping members of the organization establish a widely agreed on direction or set of purposes considered valuable for the organization; and
  2. Influence: encouraging organizational members to act in ways that seem helpful in moving toward the agreed on directions or purposes

Leithwood and Levin (2005) arrive at these two functions by adopting a definition where “the primary effect of organisational leadership would be significant change in a direction valued by the organisation”. In defending their definition or understanding of leadership, Leithwood and Levin (2005) agree that this may not be a precise definition, but that attempts to too narrowly define a complex topic like leadership is “more likely to trivialise than help bring greater clarity to its meaning”.

How the alignment project fits

Taking the emphasised phrases from the above purpose of the ALTC projects, I’d suggest/argue that the alignment project fits in the following ways:

  • leadership capacity;
    In terms of the above set of functions, the project is aiming to build into the systems and processes of each host institution the capacity to make consideration of alignment an every day part of practice. It helps establish a widely agreed direction for L&T and helps influence organisational members in moving towards that agreed direction.
  • promotion and enhancement of L&T;
    The educational literature is replete with evidence that consideration of alignment changes the conception of L&T held by academics and that it also results in improvements in student learning outcomes.
  • contemporary higher education;
    The project recognises and seeks to fulfill the increased requirements for accountability from a range of diverse source, however, it seeks to achieve it in a way that offers significant greater benefits that existing methods. As part of this the application seeks to engage with the on-going argument over centralised or devolved L&T support services by aiming for a focus on an approach to L&T support services that seeks to contextualise such support into the every day practice of teaching academics.
  • inclusiveness, diversity and collaboration; and
    The action research process suggested for use by the project is largely based on recognition within the project that engaging with the full diversity of higher education is essential. It recognises that this diversity will result in different approaches and benefits and that the process needs to enable this to happen. Similarly, collaboration is seen as essential to the project. Not just in the process used in this project, but in the aims of the project. An important aim of the project is to increase the collaboration around consideration of alignment in teaching.
  • long-term, systemic change.
    The project aims to embed consideration of alignment into the everyday practice of teaching staff. i.e. the aim of the project is long-term, systemic change. The process and approach being used is designed to achieve that aim.

The alignment project as leadership

Contemporary higher education is placing increasing importance on the concept of alignment in learning and teaching. In terms of quality assessment, program accreditation, graduate attributes and generally improving L&T alignment is broadly seen as a necessary component. However, many university courses are not all that well aligned and one explanation for this is that consideration and discussion of alignment is not a regular part of everyday teaching practice. Alignment is often only considered at the time of course and program reviews or accreditation.

The aim of the alignment project is to build leadership capacity into the system and processes of education within a university so as to encourage and enable effective and informed consideration of alignment as part of everyday teaching. This embedding of alignment into everyday practice then serves as the foundation for a range of other possibilities.

The alignment project is an example of leadership as it is attempting to encourage significant change – in the form of increased consideration of alignment at all levels – that is valued by the individual universities and the broader higher education sector. In addition, there is broad agreement in the education literature that alignment has significant positive effects on student learning outcomes.

The alignment project intends to fulfill the two indispensable functions of leadership identified by Leithwood and Levin (2005):

  • direction-setting; and
    By making considerations of alignment a visible and hopefully key aspect of everyday teaching, there should be an increased emphasis placed on learning outcomes, graduate attributes and other “outputs of learning”. This should encourage and assist academics teaching the same course or in the same program to increase discussion of these outcomes. To increase discussion of the purpose or direction of a course or program.
  • influence.
    A specific aim of the alignment project is to modify the teaching environment so that considerations, discussions and reflection upon alignment are directly encouraged and enabled. It encourages and enables them to think about how to move towards the stated directions or purpose.

Still not happy with that division. First attempt to make it concrete. Another query that arises from this is whether or not this project builds leadership capacity at two or more levels. For example, one approach is that it builds capacity at both the:

  • Institutional level; and
    At an institutional level it enables curriculum/program alignment, but also accountability etc.
  • Instructional level.
    This is the level between student and course/teacher. It makes alignment an more regular component of instruction.

Outcomes

If successful, the project should result in the consideration of alignment should becoming an every day component of teaching practice. This should/could generate the following outcomes:

  • Changes in the conceptions of L&T held by teaching staff;
    Being required to consider alignment encourages a different way of looking at teaching. This should encourage changes in the conceptions of L&T held by teaching staff towards those considered more appropriate.
  • Improvements in student learning outcomes;
    Effective considerations of alignment should increase the alignment within courses. The educational literature suggests that increases in alignment will result in improve student learning outcomes. In addition, it is likely that one of the likely additional outcomes will be making alignment more visible to students. Which should also encourage improved student learning outcomes.
  • Improvements in the quality and timeliness of quality assurance; and
    In terms of demonstrating alignment against outcomes or attributes, current quality assurance practices rely on special “mappings” that are held every few years. By embedding alignment considerations and mapping into every day teaching practice, the there is no longer any need for special “mappings”. “Mapping” information can be generated at any time as it is maintained as part of normal practice.
  • A variety of additional outcomes.
    Embedding alignment considerations as an everyday practice is the foundation of the project. The availability of “mapping” information and the on-going consideration of alignment will generate a range of additional outcomes. However, the diversity inherent in universities and higher education, combined with the fundamental change in everyday practice which this project aims to achieve means that it is unlikely we can predict successfully all of these outcomes.

Theoretical foundations

The ALTC place significant emphasis on applications having a sound and obvious theoretical base. The theoretical work that have informed my thinking about this project, and which may influence the project, include the following.

Project intent or outcome

Biggs (2001) presents an argument for a reflective institution that focuses on prospective, rather than retrospective quality. Having just re-read the paper, it seems to provide a good fit for a theory/model for the overall intent of the project.

The model is based on the idea of three aspects of AQ:

  • Quality Model.
    i.e. an espoused theory of teaching, for Biggs this is constructive alignment. For the alignment project this might be alignment a little more broadly.
  • Quality enhancement.
    A “teaching delivery system” that is designed in accordance to the quality model, i.e. one which encourages and enables alignments. In addition, the teaching delivery system should also have built-in mechanisms to continually review and improve current practice.
  • Quality feasibility.
    A process by which impediments to quality teaching are removed from the “teaching delivery system”.

The “quality model” is underpinned by a large, overlapping and diverse collection of literature from various areas including: outcomes-based quality/evaluation, instructional alignment (Cohen, 1987); curriculum alignment; graduate attributes; and, of course constructive alignment (Biggs and Tang, 2007; Biggs, 1999 ).

Understandings of universities and organisations

My personal conceptions of most organisations, but especially those like universities, are informed by Dave Snowden’s Cynefin framework and complex adaptive system.

Understandings of leadership

This is obviously an area which needs more consideration, beyond the summary given above.

Based on my limited reading, I like the description of “new leadership” (related to distributed leadership) attributed to Fullan (2008), which include:

  • respect of employees, rather than simplistic judgmentalism;
    A specific focus of this project is to help teaching staff consider the alignment of their courses. It is not to judge them. Biggs (2001) makes the point that under the type of reflective institution he outlines, the focus is on the teaching, not the teacher.
  • connecting peers with purpose and ownership;
    By embedding indications of alignment into the LMS the aim is to create connections between other teaching staff, academic leaders and also teaching support staff. The clear purpose is around considering alignment.
  • building employees’ and systems capacity; and
    This is the specific aim of the project. Building into the institution the capacity in both the employees and its systems to engage regularly in consideration of alignment.
  • transparency of practice and results.
    At the very least, the aim of this project is to enable teaching staff within the same program (i.e. group of courses/degree etc.) to see each others practice. To show what is aligned, where (or where not). It opens up the teaching practice to colleagues, hopefully in a way that is not judgmental.

Process

Based on some of the above and below, I’m leaning very much towards an action research process. The practice of L&T within a university is a complex-adaptive system. As we introduce change into the system, the system will change around us and unexpected event will happen. The type of change suggested involves a fairly widespread change in the practice of teaching academics, but also the institution. In addition, as far as I’m aware, no-one else has tried and reported on this type of change, hence it is novel. While driving towards a particular goal, we have to aim on learning as much as we can during the process.

In addition, Biggs (2001) offers the following

action research, a methodology designed precisely to generate and evaluate in-context innovations (Elliott 1991). As a result of engaging in action research, teachers change their conceptions of teaching, and teach more effectively (Kember 2000).

This type of approach also fits very closely with what is known about staff development.
i.e. the current recommendations are that staff development should be as contextualised as possible. My conception of how teaching staff would be helped to consider the alignment of their courses contains a very heavy assumption on contextualised staff development. In fact, the presence of alignment information and the transparency between courses is aimed at helping this support and development to be increasingly contextualised to the every day teaching practice of the teaching staff.

Teaching practice

One of the fundamental models/theories underpinning this project for me is the observations embodied in the following.

How academics design their teaching is not described by a rational planning model (Lattuca and Stark 2009). In part, this is because the dominant setting for academics is teaching an existing course, generally one the academic has taught previously. In such a setting, academics spend most of their time fine tuning a course or making minor modifications to material or content (Stark 2000). Academics are usually not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988).

The practice of most academics does not separate planning from implementation, and rather than starting with explicit course objectives, starts with content (Lattuca and Stark 2009). In the absence of formally documented teaching goals, the actual teaching and learning that occurs is more in line with the teacher’s implicit internalised knowledge, than that described in published course descriptions (Levander and Mikkola 2009). Formal descriptions of the curriculum do not necessarily provide much understanding about how teachers put their curriculum ideas into action (Argyris and Schon 1974)

References

Argyris, C., & Schon, D. (1974). Theory in practice: Increasing professional effectiveness. Oxford, England: Jossey-Bass.

Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University Press.

Biggs, J. (2001). The Reflective Institution: Assuring and Enhancing the Quality of Teaching and Learning. Higher Education, 41(3), 221-238.

Cohen, S. A. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16-20.

Fullan, M. (2008). The Six secrets of change. San Francisco: Jossey-Bass.

Jameson, J. (2006). Leadership in post-compulsory education: Inspiring leaders of the future. London: David Fulton Publishers.

Kember, D. (2000). Action Learning and Action Research: Improving the Quality of Teaching and Learning. London: Kogan Page.

Lattuca, L., & Stark, J. (2009). Shaping the college curriculum: Academic plans in context. San Francisco: John Wiley & Sons.

Levander, L., & Mikkola, M. (2009). Core curriculum analysis: A tool for educational design. The Journal of Agricultural Education and Extension, 15(3), 275-286.

Marshall, S. (2006). Issues in the development of leadership for learning and teaching in higher education (Occasional paper). Sydney: Carrick Institute of Learning and Teaching in Higher Education.

Stark, J. (2000). Planning introductory college courses: Content, context and form. Instructional Science, 28(5), 413-438.

Stark, J., & Lowther, M. (1988). Strengthening the Ties That Bind: Integrating Undergraduate Liberal and Professional Study. Ann Arbor, MI: Professional Preparation Project.

Southwell, D., & Morgan, W. (2009). Leadership and the impact of academic staff development and leadership development on student learning outcomes in higher education: A review of the literature. Sydney: Australian Learning and Teaching Council.

Clarification of the alignment project

Am at the University of Canberra for a second day today talking about the possibility of an ALTC grant application around alignment/curriculum mapping. The aim today is to try and clarify where the project would go and this post is an attempt to make concrete some of my thinking. The final ideas for the project may be radically different.

I’m going to use the “project clarification” section of the ALTC’s Steps in developing an evaluation plan for an ALTC grant project as the structure for this post. Mainly because of a belief that getting into the “ALTC mindset” will increase the likelihood of success.

This remains very much a work in progress. Comments and thoughts all welcome.

What is the focus of the project?

The fundamental issue the project is aiming to address is in increasing the visibility of curricula and instructional alignment in the practice of university teaching.

The observation is that consideration of alignment is not an important consideration in what most academics do around learning and teaching. Consideration of alignment is not something that students see regular reminders of in their learning. It is done and visible in places, but those times and places are not a regular, visible part of what they do.

Since it is not visible in regular practice, it is often not thought of. It doesn’t inform what they do. This is bad because there’s vast bodies of research that suggests alignment between outcomes/graduate attributes, assessment, learning activities and instructional resources is a good thing. There is evidence to suggest that many university courses do not have strong alignment. There is also evidence (McDonald, 2008) that when considerations of alignment become part of what academics regularly do as part of teaching, it encourages them to begin questioning existing practice.

The focus of the project is on how to make considerations and discussions about the level of alignment in university courses a regular and on-going part of what academics do as part of their normal teaching practices.

The aim is to make changes within the teaching context that enable and encourage these considerations and discussion of alignment to take part regularly.

What is the scope of the project?

While the project will (hopefully) be funded by the ALTC over two years, the intent is that the aim of the project is an attempt to embed these practices within the institutions. i.e. to be successful, it should be an on-going project.

Initially, the ALTC funded project will be limited to University of Canberra and CQUniversity. Participants are likely to fall into these categories:

  • Round 1 teaching staff;
    A smallish group of academic staff who will be involved in the first cycle of making alignment a regular part of their practice. The nature of this group will depend on each institution.
  • Round 2 teaching staff;
    Includes the round 1 teaching staff plus additional staff that will work with modified practices from round 1. The intent is that round 1 teaching staff will be using these practice for the second time, perhaps with the same course. The round 2 only staff will likely be chosen on the basis of having some differences (on a broad array of possible variables) with the round 1 staff.
  • learning and teaching support staff;
    These are staff tasked with helping academics improve their L&T. This might include instructional designers, staff developers etc. These staff will be working with the teaching staff during both stages to help develop appropriate approaches to integrate alignment considerations into everyday practice.
  • project leaders; and
    The staff named on the ALTC grant application will be involved in various tasks in ensuring the project is moving along and may also fulfill some of the previous roles.
  • reference group.
    A body of experts that will offer oversight and provide feedback on the projects aims, outcomes etc.

Over the 2 years of ALTC funding the project would probably aim to, at least at one of the institutions, integrated alignment into the practice of at least one program in order to test the impact at the program level. Perhaps a target of 30 or courses at each institution?

The particular approach being taken with this project is to use Moodle as the tool that enables and encourages an increased consideration of alignment.

What are the intended outcomes?

  • Indications of the impact of significant consideration of alignment on the thinking/practice of teaching staff and on the student experience.
  • Guidelines, identified challenges and suggested processes (with accompanying documentation and resources) for increasing the consideration of alignment into everyday practice.
  • Guidelines, tools and processes describing how the practice of quality assurance can change once alignment considerations are embedded into every day practice.
  • A collection of Moodle enhancements.
    These are important only in terms of the affordances they enable which can then be harnessed to improve considerations of alignment. The enhancements are likely to fall into two categories:
    1. Addition of “mapping” into Moodle.
      The ability for Moodle to enable mapping of all activities and resources against specified outcomes, attributes etc.
    2. Moodle features that use the mapping information.
      This is the really interesting part. What applications make embedding alignment into every day practice useful and interesting for the broadest array of stakeholders.

In the elevator pitch for this project I identified some of the following as propositions:

  • Curriculum maps that are generated, and continue to evolve, in the same time and space as everyday teaching and learning will have stronger validity in terms of capturing reality and consequently be of more value.
  • A learning environment that makes visible to all stakeholders the alignment (or not) of a course and then provides scaffolding necessary to improve that alignment will help improve teaching.
  • Well designed extensions to an LMS that encourage and enable improvement of course alignment will increase the quantity and quality of usage of the institutional LMS and subsequent student outcomes.
  • Building curriculum mapping around Moodle’s student “tracking” functionality will enable and encourage greater use of the student tracking functionality.
  • Properly implemented, this approach can make it easier for curriculum designers to embed assistance into the context within which teaching is taking place. If this works well, relationships will develop.

Each of these could be thought of as potential outcomes. Yes, they should be developed some more.

What are the operational processes developed to achieve the outcomes?

The process to be used in this project should lean heavily towards being adopter-focused (Surry & Farquhar, 1997). The following description is taken from Jones and Lynch (1999)

In contrast to the developer-based approach, the adopter-based approach focuses on the human, social, and interpersonal aspects of innovation diffusion (Surry & Farquhar, 1997). Developers are interested in the individual who will ultimately implement the innovation in a practical setting as the primary force for change. The adopter-based theories reject the assumption that superior products will automatically be attractive to potential adopters. They ’seek to understand the social context in which the innovation will be used and the social function the innovation will serve’ (Surry & Farquhar, 1997).

Stages in the process for this project might include:

  1. Adding mapping to Moodle.
    The focus here is on obtaining a version of Moodle that allows the mapping of every activity, resource and assignment within Moodle to a set of outcomes. i.e. the aim is to get the data stored. This is almost a purely technical process that requires little involvement from academics.
  2. Planning and first roundtable. ????
    The project groups at each of the two institutions formulate the plans/approaches they plan to use for the first round. This identifies: the participants, programs/courses, current state of mapping, approaches to be used to complete the mapping…..Have a roundtable where the reference group are shown these plans and offer advice and suggestions.
  3. Round 1
  4. Reflection/roundtable.
  5. Round 2
  6. What is the conceptual and theoretical framework underpinning the project?

    • Alignment.
    • Adopter-focused development … more buzz words.
    • Teacher conception and behaviour is a key component of quality of L&T.

    What is the context of the project?

    • 2 institutions that have adopted Moodle as enterprise LMS.
    • Higher ed sector where alignment is important in terms of graduate attributes etc. and also more broadly quality assurance and quality enhancement are increasingly more important.
    • Institutional change.

    What key values drive the project?

    • Adopter-focused.
    • Open source/open.

    References

    Surry, D., & Farquhar, J. (1997). Diffusion Theory and Instructional Technology. e-Journal of Instructional Science and Technology, 2(1), 269-278.

What are good designs/design principles for Moodle

My current institution has adopted Moodle as its institutional LMS as of 2010. Due to my role, I haven’t really had to think about how you best go about designing a Moodle course. Now, however, due to the curriculum mapping project it is likely that I am going to have to engage with this. Hence the question, what are the different principles, guidelines or approaches for designing a Moodle course site?

Why do you ask?

A part of the aim of the curriculum mapping project is to map the alignment of course activities, resources and assessment against graduate attributes, learning outcomes etc. At the moment, the project is at the stage of experimenting with existing Moodle courses – course sites that are live now – and seeing how well (or not) Moodle’s existing outcomes support can be used for this mapping purpose.

I’m only looking at a very small number of courses, however, these are courses put together by academics who care about their teaching, who want to make an effort. From this small sample it appears that, as they stand, the design of these course sites will not easily enable the clear mapping of the activities, resources and assessments against outcomes. It’s clear that the design of these courses is very different, and that’s in spite of the the institution paying some lip service to consistency of experience for the students (which is misguided I think as in the end it only results in superficial consistency and more importantly fails to engage with a key characteristic of learning and teaching – it’s diversity).

It appears that, in order for the curriculum mapping project to fully enable the mapping of activities, resources and assessments against outcomes etc, it will need to make recommendations for better course design.

Which raises the question, what is good course design in Moodle?

What are some of the possibilities?

From one perspective, it is important that this be specifically about Moodle, and not e-learning or LMS course site design in general. Moodle, like any technology, provides a set of affordances, a set of strengths and weaknesses. To get the most out of Moodle, like any technology, the design needs to be aware of the Moodle sweet spot, and its weak spots.

Moodle principles

There are some things that really work in Moodle. What are they?

An obvious place to turn is the principles underpinning Moodle itself, which are:

  1. We are all potential teachers as well as learners
  2. We learn a lot by watching others.
  3. We learn well by creating and expressing for others
  4. Understanding others transforms us.
  5. We learn well when the learning environment is flexible and adaptable to suit our needs.

Aside: would be interesting to map the content of courses with these 5 principles and find out how many follow them in some way. I think there would be surprisingly few. Following this evolution over time might be interesting as well. Do people become more informed about Moodle course design over time? Or, do they simply follow the same path they established for their first course?

Course structure or organisation

It’s my perception that the design of Moodle course sites is intended to be a sequence of sections which contain activities for students to complete. It’s interesting that one of the major “innovations” at my current institution is a “course design” that, to at least some extent, breaks this structure.

Rather than a long vertical collection of sections for each week, there is one section which breaks the course site up into a course synopsis and 5 horizontal sections – The course, resources, discussions, assessment and enquiries (which probably change depending on the course).

The courses not taking this approach seem to follow the same approach. One section as “About the course” – usually with a banner and general administration stuff – followed by the weekly sections. The content of those weekly sections is wildly different.

In part, the difference here seems to be between having lots of scrolling or not. A more typical Moodle design ends up with a couple of pages of scrolling. I’m hearing some positive responses from staff/students about the scrolling.

Is there research to see how it is received? Does the scrolling thing cause problems or benefits?

Look and feel

We’re superficial, something that looks good will often result in more immediate positive feelings, even though it’s a pain to use. A fair bit of the Moodle promotion stuff seems focused on showing that Moodle can be good looking. Even though most institutional Moodles appear to focus on consistency, rather than quality.

Learning design

More abstractly, a good course design should obviously – following the theory of alignment – be driven by the learning outcomes. With activities, resources and assessments chosen and presented in a way that best achieves those outcomes.

So, where are the good examples of good, constructively aligned Moodle course sites? What were the problem in achieving those designs?

The blended kitchen sink

One important question for the curriculum mapping project is whether or not the course site captures everything that all students experience. Where face-to-face is possible, it would appear obvious that there may well be some experiences that students have which are not captured in the course site. This suggests it won’t be captured in the mapping.

Should/can a course site contain everything, or just the online stuff?

Suggestions?

So, what say you? What are the other principles? What is out there that can inform answers to this question?

Where are the design exemplars for Moodle course sites?

At this point I’ll include a quote attributed to Stephen Downes from here

What makes e-learning effective is, of course, typically in the eye of the beholder. One person’s toast and jam may be another person’s steak and kidney pie. This is what makes the drafting of a set of guidelines for effective e-learning so difficult.

Which is just one reason why I think “one ring to rule them all” corporate approaches to web course design is a big mistake.

It’s also the reason why input from many is needed.

Suggestions from Google

Some suggestions from Google follow. Only did a single, quick search.

  • Blog post describing some alternate course formats for Moodle – related to the course structure/organisation heading above.
  • Moodle course design – a word document with good coverage of the topic
    Interesting, makes the point that a consistent theme “gives it robustness”. I like the Dave Snowden distinction between robustness and resilience. Robustness tries to prevent mistakes/failure – which with people is itself destined to fail. Whereas resilience makes it cheap to respond/solve mistakes/failure. I know which I prefer.

    For similar reasons the document advises against messing with the standard Moodle design – which is what the local “innovation” does.

Moodle curriculum mapping – Step 3

This will be a brief extension of previous work around this project. The main aim is to start identifying some of the methods used by Moodle with its current outcomes approach and how those might be harnessed and modified to support curriculum mapping. In particular, some specific questions include: What’s necessary to

  • allow the outcomes to be grouped and displayed as such when showing an activity/resource? IDENTIFIED
  • include a “help” link for each outcome or other means to explain? IDENTIFIED
  • allow the outcome scale to be used on the activity/resource to indicate how well the activity/resource meets the outcome etc? IDENTIFIED
  • display the curriculum map for a course?IDENTIFIED
  • add “outcome mapping” to those elements that currently don’t have it? IDENTIFIED
  • prevent curriculum mapping outcomes showing up in the gradebook?IDENTIFIED

This is a work in progress and will be updated over the next couple of days.

The association – where in code and the database

Moodle tracks which outcomes apply to activities and resources, the question is where in the code does this happen and where in the database is this information stored?

The code

The association appears as part of the edit screen for an activity or resource. This is implemented by moodle/course/modedit.php. This script:

  • Is given various params, including section and course, including the module being used to “edit” the activity/resource.
  • Is fairly typical PHP spaghetti code with little or no comments.
  • Acts has a harness/factory getting the module code to generate part of the form.
  • Has a section of code that retrieves and display the outcomes, all embedded in this enormous file – ugly.

The outcomes code seems to consist of (this is actually the handling of submission of the form, not display of the form – more on this below)

  • Get all the outcomes for the course (whether or not to display them, is left till then)
    if ($outcomes = grade_outcome::fetch_all_available($COURSE->id))

    fetch_all_available is implemented in moodle/lib/grade/grade_outcome.php. And basically defines a class that represents a grade outcome. fetch_all_available gets all course related outcomes listed in grade_outcomes (the detail of the outcomes) and grade_outcomes_courses (which outcomes are being used in the course).

  • Build array of grade_items
    It then loops through each outcome from above and uses moodle/lib/grade/grade_item.php to create a grade_item object for each outcome. This uses the grade_items table to store information. Am not 100% sure where this fits in.
  • The actual display is done using a “form” display…more on this below.

So the display is done using the form class defined by the module, which is an extension of moodleform_mod. As the specific modules won’t know about outcomes, the outcomes display would theoretically be done in course/moodleform_mod.php. Yep.

if ($this->_features->gradecat) {
    $gradecat = false;
    if (!empty($CFG->enableoutcomes) and $this->_features->outcomes) {

The process seems to be:

  • Get grade outcomes for the course, again
    Seems there is some duplication here, as it gets the grade outcomes for the course, all over again.
  • Get’s all grade items for the course, it any of them have an outcome set, then set this in the form?
  • A couple of other steps here, not immediately clear.

The above only seems to be preparatory. There’s a later section of code that adds the form elements for the outcomes. Again, there’s a fetch all available outcomes. This seems more directly related as it simply adds the elements.

Where does it store “mapping”

The next question is where does it store the fact that a particular activity/resource is using/assigned a particular set of outcomes?

This should be set in the code that processes the submission of the form. Which should be moodle/course/modedit.php. Ahh, this is done with grade_item as described above.

i.e. when you map an activity/resource in a course to an outcome or three, that mapping gets stored in the grade_items table. The fields in that table are (the descriptions are tentative):

  • id – the unique id for the mapping of activity/resource to a single outcome.
  • courseid – the id for the course that “owns” this mapping.
  • itemname – this is the actual name of the outcome assigned
  • itemtype – I believe this describes the type of object you’ve mapped the outcome to. Possible known values are currently mod, course.
  • itemmodule – the name of the specific module that implements the object. Possible values include: forum, bim (i.e. name of any module), assignment, resource.
  • iteminstance – I believe this is the id for this particular instance of the module. i.e. the id for the table course_modules. The pathway to more information about this instance.
  • itemnumber – for outcomes, this seems to start at 1000. It is used to give the sequence with which outcomes are assigned to the item. i.e. the first outcome assigned is 1000, the second 1001, the 3rd 1002 …. It appears that a value of 0, might indicate something important
  • iteminfo – currently set to NULL for all the entries I’ve seen so far. So, not currently sure what it is used for.
  • idnumber and calculation – also set to NULL or empty for the contents of my database – which doesn’t include a lot of real courses.
  • gradetype – integer, currently with value of 1 or 2. With outcomes I’ve set being 2.
  • grademax and grademin – fairly obvious. Seems to be set by scale and/or other stuff.
  • scaleid – the scale being used.
  • outcomeid – the id of the outcome
  • gradepass multfactor plusfactor aggregationcoef – various factors used for grade calculation, I believe.
  • sortorder – different integer values – purpose not immediately obvious.
  • display – big int, currently all set to 0. Not sure of purpose.
  • decimals hidden locked locktime needsupdate – various flags ?
  • timecreated timemodified – time stamps. Could be useful for identifying outcomes that need to be re-checked.

It appears that grade items and outcome items are treated the same, hence their use of the same table. The full view of categories and items give a good overview of this table.

There is the concept of categories of grades/items, this might be one avenue. i.e. a category for curriculum mapping.

What is the implication of this?

The next question is what are the implications to the rest of Moodle. If I map all the activites/resources within a course against a complex set of outcomes, does it have an effect on the gradebook? Any where else?

So, I’ve set outcomes for a number of activities/resources in a course. Does this show up anywhere else? Two ways of looking:

  • Check the gradebook from web interface.
  • Look for use of grade_item class/object.

Mmmm, not good. It appears that every time you add an outcome, it get’s added to the gradebook. In terms of curriculum mapping, not what is desired. This is perhaps the first obvious example that curriculum mapping and tracking student performance, while to some extent similar, serve different purposes.

The column in the gradebook for each outcome that is added, has a header that is a link. The link is to a script that shows some detail of the resource or activity that it the outcome is associated with.

Need to turn this off.

Now, you can hide an element in the gradebook. But that just greys it out, doesn’t remove it entirely from consideration, which is what is wanted here.

Adding a description/help

Problem

At least initially the outcomes etc shown are not going to make much sense to a teacher. Moodle currently only displays the name of the outcome. The teacher would have to somewhere else to read up on the outcome before they can determine if it applies. It would be helpful if additional assistance was provided there and then.

Some options in terms of what could be displayed, include:

  • The description of the outcome.
    As it stands Moodle allows each outcome to have a textual description. Displaying this as a roll-over or in a new window could provide a minimal level of assistance.
  • Link to institutional area for discussion and description of outcomes.
    The assumption being that most institutions would have a website in which institutional outcomes etc are discussed or described. Providing a link to this area, especially to the context specific to the a particular outcome might be useful.
  • Link to other examples.
    Many of the forms of outcomes etc. are likely to be used in other courses. e.g. institutional graduate attributes. It might be useful to give the option to see other examples of how these attributes are used in other courses. Even to the extent of link directly to those courses and/or reflections/discussion from other teachers using this outome.

These ideas range from the simple and static through to something you’d want to have some curating.

Possible solutions

The outcomes are displayed around line 220 in moodle/course/moodleform_mod.php. This is where the change would have to happen. Some possibilities include:

  • Using a Moodle helpbutton.
    Moodle forms have a function – setHelpButton – which associates a help button with an element. Very easy to make this modification. However, the problem is that the helpbutton is typically a call to open a new, small browser window to display HTML file.

    This is problematic as the outcomes are added via the Moodle interface and doesn’t provide support for adding a help file. So, outcome specific would be difficult. However, an institutional area/approach could be possible. It would require the institution to create HTML files for each outcome.

    Let’s do a simple test, put the Moodle code under git so I can manage this. And add a help button for each outcome. As expected, works easily. There is the question of how to create the filename for the HTML file. Most outcomes will have spaces and other characters that don’t necessarily play nicely with a filename. The language translation side of Moodle could help there, convert the complete outcome name into something more file system friendly.

  • More complicated HTML
    Another approach would be to add roll overs, additional links etc to the outcomes. This would require a more radical modification of the Moodle core but not much more than the above. Not to mention the desire to separate attributes up into groups.

Groupings of outcomes

Problem

It is likely that a course may have multiple different types of outcomes etc to map against. e.g institutional graduate attributes, discipline graduate attributes, course learning outcomes, program learning outcomes etc. There are two possible solutions (possibly complementary):

  1. Show outcomes grouped by category.
    To allow the mapping of an activity/resource against all these different groupings, it would be useful to separate out the different outcomes by category. So you could have a visible separation.
  2. Have a separate cross mapping.
    Mapping against all of these different outcomes might be somewhat tiresome, especially given a large amount of overlap between them. An approach that has been used is to produce a mapping between the different outcomes and a single point, and then only map activities/resources against that single point. Which of the different outcomes applies, can then be derived from the single point.

Possible solutions

Showing outcomes by category is going to need:

  • Some way of specifying categories/groups of outcomes.
    Which probably implies an additional database table and an additional interface or modification to an existing interface (e.g. the import outcome process) to specify which category an outcome belongs to.

    A separate interface minimises changes to core Moodle code, modifying existing interfaces is probably a more user friendly approach, depending on how widespread this need is.

  • Modification to the form display to recognise the categories.
    This should/would be a fairly simple thing to do, given the information above and Moodle’s form library.

    Let’s try a simple test. Create two boxes of outcomes that contain a copy of the same outcomes. Mainly to test if nested header/boxes work. No, they don’t. You’d have to use a separate header label and then have separate boxes for each, perhaps a table? Though Moodle dislikes table for layout…..

Display a curriculum map for a course

Problem

One of the basic functions for curriculum mapping is to get a report that shows how widely (or not) the outcomes are represented within the course in terms of resources, activities and assessments. i.e. you want a visual representation of the outcome mappings.

Possible solutions

Well, it’s basically a report, but you might want it more interactive than that.

Well, the existing outcomes report can do this to some extent. So an extension of this, or the additional of a mapping report might fill the bill.

To a large extent this is a fairly standard web application. Get the data from the database and display it in an appropriate form.

You’ll be needing data from the following tables:

  • grade_items – given a course id, this will give you all the outcomes for the course that have been mapped to activities/resources and the ids of those activities and resources.
  • grade_outcomes – will give you details about the outcomes – name, description, and scale id.
  • scale – details about the scales
  • course_modules – more information about the module/activity, most importantly perhaps the section of the course in which it appears.
  • resource – for the same reason as modules

Show outcome scale on activity/resource

Problem

Rather than simply “mapping” an outcome to a particular activity/resource, it may be useful to indicate how well/to what level does the activity/resource map to the outcome. i.e. use a scale, rather than a simple check box.

This is a fairly major distinction between outcomes for curriculum mapping and outcomes for student progress.

Possible solutions

It’s looking like a separate set of “Mapping outcomes” might be the way to go. This would also get around the problem with the gradebook from above. This might mean duplicating the items table, or at least adding a flag to differentiate between mapping and progress outcomes.

Similarly, could probably still use the standard outcomes “creation/import” process for both purposes.

Adding separate support would also help make it a bit easier to add curriculum mapping to an instance of Moodle by minimising disruption to the Moodle core.

Elements that don’t have outcomes

Problem

As outlined earlier there are some elements of a Moodle course site to which you can’t map outcomes. The outcomes don’t appear on the “edit” page. Those identified so far are labels and sections.

Sections might be useful, if you wanted to map a course by weeks, rather than by item. But perhaps not, you can generate such a map by aggregating the mapping of the contents.

Labels are way of inserting HTML into sections. Currently they don’t have support for outcomes. I’ve already seen in one course how such labels can be used to specify tasks, such as reading.

Possible solutions

Well, labels are the only real problem. The form for labels is generated using moodle/course/modedit.php. The same for anything else. It is the place where outcomes are shown. So, perhaps it’s just a switch that needs setting. Perhaps, outcomes aren’t here as it isn’t expected that these will be used in grades – i.e. student progress tracking.

Nope. The mod_form.php file for label actively turns off outcomes in a setting. Yep, set that to true and outcomes are there.

In light of the above, you’d probably have a separate set of outcomes for mapping, have this defined as a feature that modules can turn on/off and go down that route.

Elevator pitch for “Moodle curriculum mapping”

Over the next few weeks I am going to have a few meetings in which I’ll need to explain and justify the curriculum mapping project. This post is an attempt to further develop and share some of the perspectives and to develop an elevator pitch for the project.

It’s also reinforced the need to develop a name for the project. “Moodle curriculum mapping” doesn’t really send the right message.

Aside: I’m somewhat skeptical of the value and possible side effect of an aligned curriculum and curriculum mapping in general. (e.g. I’m not convinced institutional graduate attributes are possible or sensible, I’m not convinced that a mapped curriculum gives a true representation of what the student will actually learn/experience). However, I can see that this is becoming very important within Australian universities and that it will be done. I’m hoping this project can take the implementation of curriculum mapping in more interesting and useful directions. Time will tell.

Elevator pitch

The elevator pitch is formed by the sub-headings of the specific aims section. The “specific aims” section was written first.

At its simplest, the project aims to build on existing practices around curriculum mapping and fulfill existing needs, such as accreditation. More importantly, the project aims to make learning outcomes, graduate attributes etc a part of the everyday practice for a majority of academics in a way that is useful. As a result, it should help improve the validity of curriculum maps, encourage greater quantity and quality of use of the LMS and show how the institution is leading the sector. Most importantly, the project aims to provide a foundation that enables CDDU to more effectively engage with academics and, in broader terms, enable and encourage improvements in teaching and student learning outcomes.

It will do this through modifications to the Moodle LMS that aim make learning outcomes, graduate attributes and the alignment of those with learning activities, resources and assessment a “first class objects” within Moodle. These modifications, and more importantly, the processes used to roll them out will be designed to encourage and enable adoption and behaviour change.

Difficulties

The difficulties associated with doing this is that my overall focus for the project is around improving learning and teaching, which is a complex task with lots of connections. When people talk about curriculum mapping, they are typically talking about many different things, from many different perspectives.

Even more difficult is the fact that I am approaching this from the perspective of intervening in a complex adaptive system. This means that I don’t (and I believe, I can’t) have a firm idea of exactly where this project is going to go. This is because any fixed outcome is a waste of time, as the project proceeds we will learn more and the system (i.e. the university, its staff and their practice of L&T) will change around the project. What we think is a good thing to do in 6 months time, will be very different from where what we think would be good now.

And this is exactly the sort of “meta-discussion” that many of the people I’m going to talk to, will hate.

Some specific aims

So, let’s start with some specific aims.

Terminology:

  • “outcomes etc.” – used to encapsulate the broad array of “stuff” people want to map, including: university graduate attributes, course learning outcomes, discipline outcomes/attributes etc.
  • “course” – the smallest unit of study in which a student enrolls. Lasts for around 12 weeks.
  • “program” – a collection of courses that form a degree.
  • “majority” – the following often makes statements like “the majority of academic staff don’t teach well”. This is not a deficit model of academic staff. In the vast majority of cases the “don’t teach well” is due to contextual issues. I don’t blame the teacher.

    There are a small number of academic staff who teach very well. They typically do this in spite of contextual issues and because they are intrinsically motivated. This group are in the minority.

    My interest is in helping change the context so that the majority of academic staff are enabled and encouraged to improve their teaching.

Practical outcomes, reuse and a foundation

Just about every program that needs to be accredited has to generate some form of curriculum map, usually to meet the requirements of the external accrediting body. Increasingly, Australian universities are being required to demonstrate the presence and use of graduate attributes, typically illustrated through curriculum maps.

Proposition: This project will provide the functionality required to generate these and other curriculum maps.

There is existing work being done to generate these curriculum maps using established approaches (i.e. Word documents and spreadsheets). Rather than waste this work, these need to be used as inputs into this process.

Curriculum mapping is not a ends unto itself. It is typically part of a process used to increase understanding of a course and in particular its alignment and its relationships with other courses. That improved understanding informs subsequent action.

The intent is that this project is not simply about curriculum mapping, but it is focused on how the project can provide a foundation to enable and encourage subsequent action.

Make it part of everyday practice

Curriculum mapping is based around the idea that having alignment between the outcomes etc. and the learning activities, resources and assessments within a course is a good thing. The trouble is that as currently implemented outcomes etc are not part of the everyday practice for academic staff. Most academic staff, when planning a course, don’t think about outcomes etc. and alignment.

Support for this perspective comes from one of the few empirical examinations of academic practice that I’m aware of, the work of Stark, Lowther et al (1988), Stark et al (1990), Stark (2000), and Lattuca and Stark (2009). Some choice quotes from my thesis drawing on this work

How academics design their teaching is not described by a rational planning model (Lattuca and Stark 2009). In part, this is because the dominant setting for academics is teaching an existing course, generally one the academic has taught previously. In such a setting, academics spend most of their time fine tuning a course or making minor modifications to material or content (Stark 2000). Academics are usually not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988). The practice of most academics does not separate planning from implementation, and rather than starting with explicit course objectives, starts with content (Lattuca and Stark 2009).

In part, this is because consideration of outcomes etc. are not a part of what many of them do around learning and teaching. That is, the LMS they are using doesn’t usually provide any support or recognition of outcomes etc. It’s not part of what they do in face-to-face L&T. To some extent, it’s a case of out of sight and out of mind.

This problem is made worse in institutions where an increasing number of courses are being taught multiple times a year. In my local context, this means that rather than an academic designing and teaching the course and its assessment. The design is done by the person currently teaching the course, and the next person teaching the course has to live with that design.

This becomes an increasing problem when many of these staff are contract staff employed to deliver the course. They don’t design the course, so the only place they see the outcomes etc. are in the small section of the course profile/syllabus.

Proposition: If alignment between outcomes etc, learning resources, activities and assessment is a good thing, then making outcomes etc and their relationship with learning resources, activities and assessment a highly visible and first class component of the LMS/learning environment is necessary to increase alignment, or at least consideration of it.

Improve the validity of the mappings

As argued above, outcomes etc and alignment is not a key component of the thinking of most academics. It has been widely recognised for some time that the validity of the mappings represented in static documents is somewhat questionable. Some more thesis quotes

In the absence of formally documented teaching goals, the actual teaching and learning that occurs is more in line with the teacher’s implicit internalised knowledge, than that described in published course descriptions (Levander and Mikkola 2009). Formal descriptions of the curriculum do not necessarily provide much understanding about how teachers put their curriculum ideas into action (Argyris and Schon 1974).

Curriculum mapping is typically done in a way divorced in time and space from the context within which academics teach. Subsequently, due to the known limitations of human memory – not to mention pressure to comply – it is unlikely that such curriculum maps capture the full complexity what occurs in a course. Given the static nature of such maps and their lack of use in everyday teaching, then over time the validity of their representation is only going to decrease.

Proposition: Curriculum maps that are generated, and continue to evolve, in the same time and space as everyday teaching and learning will have stronger validity in terms of capturing reality and consequently be of more value.

Enable and encourage improvements in teaching

The mapping process is fairly straight forward, if you are familiar and comfortable with a lot of the educational language involved in the process. A lot of academics aren’t. Some more thesis quotes

In the absence of formal qualifications or knowledge in learning and teaching, most academics teach in ways they have been taught (Phillips 2005) and/or which fit with disciplinary norms and their recent teaching experience (Entwistle 2003). Academic staff rarely read educational literature or call upon any available expert assistance when planning a course (Stark 2000).

A well designed mapping process would provide the scaffolding necessary for academic staff to be guided through the mapping process. It would provide pointers to similar maps, explanations of why it was done a certain way, reflections and insights from other teaching academics (not instructional designers) etc.

More importantly, the mapping process is the educationally easiest part of this process. What’s much more difficult is, once you have the map, using that to improve the alignment of the course learning resources, activities and assessments. Knowing how best, within the specific context, to provide students with the opportunity to practice and receive feedback on “critical thinking” is far more difficult than identifying where it is or isn’t provided.

The aim here is not just to help academics map the course, but then build on this to enable and encourage them to improve their courses.

Proposition: A learning environment that makes visible to all stakeholders the alignment (or not) of a course and then provides scaffolding necessary to improve that alignment will help improve teaching.

Encourage greater usage (quantity and quality) of institutional LMS

A lot is written about the poor quantity and quality of the learning and teaching that occurs within an institutional LMS. The vast majority of course sites are little more electronic photo-copiers, places to disseminate text. This is due to a variety of issues, most of them contextual and nothing to do with the LMS. However, some are due to the nature of the LMS and the types of tasks it makes easy (e.g. uploading some powerpoint slides) and the types of tasks for which it has no support (making visible and offering advice on how to improve course alignment).

Proposition: Well designed extensions to an LMS that encourage and enable improvement of course alignment will increase the quantity and quality of usage of the institutional LMS and subsequent student outcomes.

In a more institutional specific aspect of this aim, is the observation that Moodle (my institutions LMS) already offers support for tracking student progress against outcomes. However, this feature, which is optional, is not even enabled within the institutional instance of Moodle, and is currently not being used. Even though, there are parts of the institution that want to use this sort of feature.

Proposition: Building curriculum mapping around Moodle’s student “tracking” functionality will enable and encourage greater use of the student tracking functionality.

Demonstrate innovation and leadership

Within my own institution I have heard Deputy Vice-Chancellors ask “Where is all the innovative learning and teaching? We used to be at the fore-front.”
I have seen universities claim how innovative they are being around curriculum mapping because they are moving from using a Word document as the course curriculum map, to using an Excel spreadsheet. It’s not hard to be innovative in this area.

Proposition: The ideas described here are innovative and if successfully implemented can enable the institutions involved to demonstrate leadership within the sector.

Enable the CDDU to engage more effectively with academics

I work for the Curriculum Design and Development Unit (CDDU) at CQU. While I am not a curriculum designer, curriculum design is a key part of what the unit does. For me, the overall aim of curriculum design is to help academics improve the quality of their teaching and their students learning.

Current approaches to curriculum design have little impact. Arguably, this is because of a number of reasons, including – but not limited to:

  • Mismatch between the instructional design process and how academics plan courses.
    As outlined above, the majority of academics don’t use a rational planning model for course design. Instructional design is typically guided by a rational planning model. This mismatch is incredibly difficult for most academics to bridge, or even understand. The mismatch, in many cases, limits outcomes.
  • Help arrives outside of the context of need.
    An academic usually has a problem or question about teaching, while they are planning or teaching (which as argued above, is typically not separate tasks for academics). Instructional design assistance is typically not available within this context of need. Instead the academic must remember their need, at a separate time remember to ask for assistance, and then try to remember and explain the context of need to the instructional designer. Is it any wonder academics don’t draw on expert help?
  • Instructional designer as the police or fire brigade.
    The only time you need the police or fire brigade is when you are in trouble. The association people build up of these services is, trouble. Academics can form a similar impression of instructional designers, we only see them when I have a problem. It’s made worse in cases when management directs the academic to see the instructional designer.

Proposition: Properly implemented, this approach can make it easier for curriculum designers to embed assistance into the context within which teaching is taking place. If this works well, relationships will develop.

Specific project stages

While the specifics of the project are up in the air, not the least because of contextual uncertainties, it is possible to identify a collection of likely project stages:

  • Explore what is possible with Moodle.
    Where I am now. Playing with Moodle and its current outcomes functionality to see how it works and where the limits are. To find out what might need to be done, what are the limitations.
  • Talk with external partners.
    In mid-April I’m going to Canberra to talk with folk at University of Canberra and find out what there interest is around this topic.
  • An initial local trial.
    Half in this at the moment, but more progress once the “exploration” stage is complete. Work with a couple of programs to get some initial “mappings” done. This might involve a bit of rough coding to enable simple practices. Again, getting more of an idea of the project and what needs to happen. The purpose of this stage is to generate something concrete to show people what we’re talking about, rather than rely on abstract hand waving.
  • Some “innovative” applications.
    Arising out of the last stage, the aim will be to generate some new applications out of the initial trial to illustrate what might be possible. To really show that this isn’t just about curriculum mapping.
  • Initial publications
    Around about this stage we should be in a place for an initial publication or two, to get the word out.
  • An ALTC grant application.
    This is by July this year.
  • Contributions to the Moodle community.
    Eventually, when/if we produce something useful. It has to be given back to the broader community.

References

Lattuca, L. and J. Stark (2009). Shaping the college curriculum: Academic plans in context. San Francisco, John Wiley & Sons.

Stark, J., M. Lowther, et al. (1988). “Faculty reflect on course planning.” Research in Higher Education 29(3): 219-240.

Stark, J. and et al (1990). Planning introductory college courses: Influences on faculty. Ann Arbor, National Center for Research to Improve Postsecondary Teaching and Learning.

Stark, J. (2000). “Planning introductory college courses: Content, context and form.” Instructional Science 28(5): 413-438.

Moodle curriculum mapping – Step 2

This is the second exploration of an idea for enhancing Moodle to enable curriculum mapping. It carries on from the first step and is part of a broader project.

The aim today is to:

  • Create a CSV file of Moodle outcomes for a couple of programs.
    Mostly to get a feel for the outcomes that accrediting bodies are after and to test out this “uploading” of outcomes. Also to get some insight into how the “scales” might work.
  • “Map” a course or two with those outcomes.
    The aim is to get a feel for how difficult doing this actually is and how well it works. Perhaps get some insights into ways it could be made easier/more effective.
  • Start identifying the database structures where that information is placed.
    This is a pre-cursor to starting to develop extensions to Moodle that will draw on this information. It helps identify where the information is, what is there and what might be possible in terms of development.

Am going to be updating this post throughout today (30 March, 2010)

Moodle outcomes CSV file testing

Moodle allows you to upload outcomes into Moodle via a CSV file. The format is a 6 field CSV file

  • outcome_name – full name
  • outcome_shortname – short name
  • outcome_description
  • scale_name – name of scale
  • scale_items – comma separate list of scale items
  • scale_description

e.g.

outcome_name;outcome_shortname;outcome_description;scale_name;scale_items;scale_description
Participation;participation;;Participation scale;”Little or no participation, Satisfactory participation, Full participation”;

Each outcome in Moodle is associated with a scale. It’s typically used to make student performance against the outcome. For curriculum mapping, I believe the scale can be used to measure how well the course/activity/resource meets the outcome/attribute etc.

The task now is to create a useful CSV outcomes file for my purposes. The choices that exist include:

Am thinking I’ll start with the institutional graduate attributes – mostly for political reasons – and then do one of the disciplinary bodies outcomes for a bit more learning.

Graduate attributes

Not 100% certain this represents the current state of the institution’s graduate attributes, but it’s good for an experiment. The institution is apparently introducing graduate attributes progressively during 2010 with all undergraduate programs done from Jan 2011 and all other programs from 2012.

The institution has 8 graduate attributes:

  • Communication
  • Problem solving
  • Critical thinking
  • Information literacy
  • Team work
  • Information technology competence
  • Cross cultural competence
  • Ethical practice

As it stands, I’ve been unable to find any description of these. However, a document describing the project has developed some “levels of achievement” for the attributes and offered descriptions of those levels using learning outcomes and the revised Bloom’s taxonomy.

The three levels are: introductory, intermediate and graduate. Each of the outcomes/levels are associated with learning domains from the revised Bloom’s taxonomy.

Note: my aim here is to identify what has been done and work out how it can be translated into Moodle’s outcomes CSV file. Not to judge what’s been done.

The CSV file

The first version of a CSV file for the attributes is done and successfully in. Will reflect more on this after lunch.

The Moodle help documentation suggests that the format is as listed above, with outcome_description and scale_description as optional. That means that you don’t have to include them in a line, but you do need to include all fields. Getting the format exactly right was an interesting experience in trial and error.

The first two lines of the file are

Communication;comm;”Described here http://dmai.cqu.edu.au/FCWViewer/view.do?page=7949″;”CQU Graduate Attributes (Communication)”;”Introductory – Use appropriate language to describe/explain discipline-specific fundamentals/knowledge/ideas (C2), Intermediate – Select and apply an appropriate level/style/means of communication (C3), Graduate – Formulate and communicate views to develop an academic argument in a specific discipline (A4)”;

What is looks like

When trying to map an activity/resource in Moodle, you use the “edit” facility for that activity/resource and a part of the resulting page looks like the following – click on it to see it bigger.

Moodle outcomes

Some comments on this image:

  • Duplicate outcomes suggest Outcome management not great.
    You can see three outcomes for Communication. This is due to the problems associated with importing the CSV file – 2 failed attempts, followed by a successful one. And subsequent difficulties in finding out how to delete the older versions of the outcome…..Ahh, you have to go to “edit outcomes”.
  • Not enough information.
    While it wouldn’t be a problem eventually, the problem I’m currently facing is that I’m not familiar enough with the outcomes to understand what they mean. I want some additional pointers in the interface – even just the normal Moodle help link (a little question mark). This absence is somewhat related to the next point.
  • Can’t use the scale here and now.
    For curriculum mapping, I want to select the scale here and now. The idea is to specify to what level this activity/resource meets the outcome. This highlights the difference in purpose between the outcomes in Moodle (focused on measuring individual student performance) and what the outcomes would be used in many forms of curriculum mapping (mapping how well a course covers outcomes). For Moodle outcomes the scale starts to apply in the gradebook, i.e. when you’re marking the individual student. Not in the activity/resource.

    Graduate attributes could be used for both approaches, map the course and also track student progress.

  • The need for groupings of outcomes.
    The first outcome “David’s first outcome” is some from some earlier testing. But it does highlight an additional requirement, the ability to separate (and perhaps map) between different groupings of outcomes. e.g. CQU’s graduate attributes, course learning outcomes and perhaps discipline accrediting body learning outcomes.
  • The Moodle workflow is somewhat limited.
    With outcomes, as with other aspects of Moodle, the “workflow” – the sequence of screens you go through as you perform a task – leaves a bit to be desired. It’s not often clear where to go, or as you finish how best to proceed.

Other outcomes

Am now looking at the accreditation requirements for psychology and public relations to understand what is there and what implications that might have for this idea.

In terms of public relations it appears to be a combination of course outcomes, university graduate attributes and some specific “criteria/areas” specified by the program.

In psychology, there’s an odd mixture of discipline specific “graduate attributes”, with each having its own set of critiera, and a collection of “skills” to “map” assessment against.

Where’s the data?

Seems the outcomes stuff might be stored in three tables:

  • grade_outcomes: id, courseid, shortname, fullname, scaleid, description, timecreated, time modified, usermodified
    Obviously the table the CSV import modifies.
  • grade_outcomes_courses: id, courseid, outcomeid
    Links a course with an outcome in the previous.
  • grade_outcomes_history: id, action, oldid, source, timemodified, loggeduser, courseid, shortname, fullname, scaleid, description.
    Not sure on this one.

So, one question is where does the mapping against a particular activity/resource get put?

What about code?

moodle/lib/grade/grade_outcome.php defines a class grade_outcome, that is meant to handle it all, including database manipulation.

From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques

The following draws on principles/theory from psychology to guide thinking about how to incorporate “data” from “academic analytics” into an LMS in a way that encourages and enables academic staff to improve their learning and teaching. It’s based on some of the ideas that underpin similar approaches that have been used for students such as this Moodle dashboard and the signals work at University of Purdue.

The following gives some background to this approach, summarises a paper from the psychology literature around behaviour modification and then explains one idea for a “signals” like application for academic staff. Some of this thinking is also informing the “Moodle curriculum mapping” project.

Very interested in pointers to similar work, suggestions for improvement or expressions of interest from folk.

Background

I have a growing interest in how insights from psychology, especially around behaviour change can inform the design of e-learning and other aspects of the teaching environment at universities in a way to encourage and enable improvement.
Important: I did not say “require”, I said “encourage”. Too much of what passes in universities at the moment takes the “require” approach with obvious negative consequences.

This is where my current interest in “nudging” – the design of good choice architecture and behaviour modification is coming from. The basic aim is to redesign the environment within which teaching occurs in a way the encourages and enables improvement in teaching practice, rather than discourages it.

To aid in this work, I’ve been lucky enough to become friends with a pyschologist who has some similar interests. We’re currently talking about different possibilities, informed by our different backgrounds. As part of that he’s pointing me to bits in the psychological literature that offers some insight. This is an attempt to summarise/reflect on one such paper (Michie et al, 2008)

Theory to intervention

It appears that the basic aim of the paper is to

  • Develop methods to clarify the list of behaviour change techniques.
  • Identify links between the behaviour change techniques and behavioural determinants.

First a comparison of two attempts at simplifying the key behavioural determinants for change – the following table. My understanding is that there are some values of these determinants that would encourage behaviour change, and others that would not.

Key Determinants of Behaviour Change from Fishbein et al., 2001; Michie et al., 2004
Fishbein et al Michie et al
Self-standards Social/professional role and identity
Knowledge
Skills Skills
Self-efficacy Beliefs about capabilities
Anticipated outcomes/attitude Beliefs about consequences
Intention Motivation and goals
Memory, attention and decision processes
Environmental constraints Environmental context and resources
Norms Social influences
Emotion
Action planning

It is interesting to see how well the categories listed in this table resonate with the limits I was planning to talk about in this presentation. i.e. it really seems to me, at the moment, that much of the environment within universities around teaching and learning is designed as to reduce the chance of these determinants to be leaning towards behaviour change.

Mapping techniques to determinants

They use a group of experts in a consensus process for linking behaviour change techniques with determinants of behaviour. The “Their mapping” section below gives a summary of the consensus links. The smaller headings are the determinants of behaviour from the above table, the bullet points are the behaviour change techniques.

Now, I haven’t gone looking for more detail on the techniques. The following is going to be based solely on my assumptions about what those techniques might entail – and hence it will be limited. However, this should be sufficient for the goal of identifying changes in the LMS environment that might encourage change in behaviour around teaching.

First, let’s identify some of the prevalent techniques, i.e. those that are mentioned a more than once and which might be useful/easy within teaching.

Prevalent techniques

Social encouragement, pressure and support

The technique “Social processes of encouragement, pressure, support ” is linked to 4 of the 11 determinants: Social/professional role and identity, beliefs about capabilities, motivation and goals and social influences. I find this interesting as it can be suggested that most teaching is a lone and invisible act. Especially in a LMS where what’s going on in other courses. Making what happens more visible might enable this sort of social process.

There’s also some potential connection with “Information regarding behaviour of others” which is mentioned in 3 of 11.

Monitoring and self-monitoring

Get mentioned as linked to 4 of 11 determinants. Again, most LMS don’t appear to give good overall information about what a teacher is doing in a way that would enable monitoring/self-monitoring.

Related to this is “goal/target specified”, part of monitoring.

There’s more to do here, let’s get onto a suggestion

One suggestion

There’s a basic model process embedded here, something along the lines of:

  • Take a knowledge of what is “good” teaching and learning
    For example, Fresen (2007) argues that the level of interaction, facilitation or simply participation by academic staff is a critical success factor for e-learning. There’s a bunch more literature that backs this up. And our own research/analysis has backed this up. Courses with high staff participation show much higher student participation and a clearer correlation between student participation and grade (i.e. more student participation, the higher the grade).
  • Identify a negative/insight into the behavioural determinants that affect academic staff around this issue.
    There are a couple. First, it’s not uncommon for staff to have an information distribution conception of teaching. i.e. they see their job as to disseminate information. Not to talk, to communicate, or participate. Associated with this is that most staff have no idea what other staff are doing within their course sites. They don’t know how often other staff are contribution to the discussion forum or visiting the course site.
  • Draw on a behavioural technique or two to design an intervention in the LMS that can encourage a behaviour change. i.e. that addresses the negative in the determinants.
    In terms of increasing staff participation you might embed into the LMS a graph like the following. Embed it in such a way as the first thing an academic sees when they login, is the graph – perhaps on part of the screen.

    Example staff posts feedback

    What this graph shows is for a single (hypothetical) staff member the number of replies they have made in course discussion forums for the three courses the staff member has taught. The number of replies is shown per term, in reality it might be shown by week of term – as the term progresses.

    This part can hit the “monitoring”, “self-monitoring” and “feedback” techniques.

    The extra, straight line represents the average number of replies made by staff in all courses in the LMS. Or alternatively, all courses in a program/degree into which the staff member teaches. (Realistically, the average would probably change from term to term).

    This aspect hits the “social processes of encouragement, pressure, support”, “modelling/demonstration behaviour of others”. By showing what other people are doing it is starting to create a social norm. One that might perhaps encourage the academic, if they are below the average, to increase their level of replies.

    But the point is not to stop here. Showing a graph like this is simple using business intelligence tools and is only a small part of the change necessary.

    It’s now necessary to hit techniques such as “graded task, starting with easy tasks”, “Increasing skills: problem-solving, decision-making, goal-setting”, “Planning, implementation”, “Prompts, triggers, cues”. It’s not enough to show that there is a problem, you have to help the academic with how to address the problem.

    In this case, there might be links associated with this graph that show advice on how to increase replies or staff participation (e.g. advice to post a summary of the week’s happenings in a course each week, or some other specific, context appropriate advice). Or it might also provide links to further, more detailed information to shed more light on this problem. For example, it might link to SNAPP to show disconnections.

    But it’s even more than this. If you wanted to hit the “Environmental changes (e.g. objects to facilitate behaviour)” technique you may want to go further with than simply showing techniques. You may want to enable this “showing of techniques” to be within a broader community where people could comment on whether or not a technique worked. It would be useful if the tool help automate/scaffold the performance of the task, i.e. moved up the abstraction layer from the basic LMS functionality. Or perhaps the tool and associated process could track and create “before and afters”. i.e. when someone tries a technique, store the graph before it is applied and then capture it at sometime after.

It’s fairly easy to see how the waterfall visualisation (shown below) and developed by David Wiley and his group could be used this way.

education,data,visualization

Their mapping

Social/professional role and identity

  • Social processes of encouragement, pressure, support

Knowledge

  • Information regarding behaviour by others

Skills

  • Goal/target specified: behaviour or outcome
  • Monitoring
  • Self-monitoring
  • Rewards; incentives (inc. self-evaluation)
  • Graded task, starting with easy tasks
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Rehearsal of relevant skills
  • Modelling/demonstration of behaviour by others
  • Homework
  • Perform behaviour in different settings

Beliefs about capabilities

  • Self-monitoring
  • Graded task, starting with easy tasks
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Coping skills
  • Rehearsal of relevant skills
  • Social processes of encouragement, pressure and support
  • Feedback
  • Self talk
  • Motivational interviewing

Beliefs about consequences

  • Self-monitoring
  • Persuasive communication
  • Information regarding behaviour, outcome
  • Feedback

Motivation and goals

  • Goal/target specified: behaviour or outcome
  • Contract
  • Rewards; incentives (inc. self-evaluation )
  • Graded task, starting with easy tasks.
  • Increasing skills: problem-solving, decision-making, goal-setting
  • Social processes of encouragement, pressure, support
  • Persuasive communication
  • Information regarding behaviour, outcome
  • Motivational interviewing

Memory, attention, decision processes

  • Self-monitoring
  • Planning, implementation
  • Prompts, triggers, cues

Environmental context and resources

  • Environmental changes (e.g. objects to facilitate behaviour)

Social influences

  • Social processes of encouragement, pressure, support
  • Modelling/demonstration of behaviour by others

Emotion

  • Stress management
  • Coping skills

Action planning

  • Goal/target specified: behaviour or outcome
  • Contract
  • Planning, implementation
  • Prompts, triggers, cues
  • Use of imagery

References

Fresen, J. (2007). “A taxonomy of factors to promote quality web-supported learning.” International Journal on E-Learning 6(3): 351-362.

Michie, S., M. Johnston, et al. (2008). “From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques.” Applied Psychology: An International Review 57(4): 660-680.

First step in “Moodle curriculum mapping”

This is perhaps the first concrete step in a project that is aiming to look at how the act of curriculum mapping can be embedded into the, increasingly, most common task and tool used by academics. That is, how can an LMS (like Moodle) be used/modified to make curriculum mapping a part of what academics do, both in terms of maintaining the mapping, but more importantly using the mapping in interesting and useful ways.

As outlined in a previous post it appears that Moodle (the institutional LMS at my current institution) already has functions that offer some basic level of support for curriculum mapping. However, they are mostly used/intended for tracking student outcomes/performance. This post documents an initial foray into using these functions to implement some form of curriculum mapping. The plan is:

  • Use existing functions to map a course or two and find out how that works and how it might be made better.
  • Use the data of the mapping to generate some applications that use the data.

Turned out, due to having to fight other fires, that today’s work was limited. Only small progress.

The courses and the people

I’m working with 2/3 courses. Two from in and around public relations and one from psychology. More detail on these later.

The set up

The plan is to perform this project on a copy of Moodle running on my laptop. i.e. it’s separate from any systems people rely upon and allows me the freedom to code. I’ll be taking backups of the live course sites for the above course, restoring them on my laptop’s Moodle and mapping the courses.

My first problem was to restore the backups. I had an old version of libxml which meant the restore process in Moodle wasn’t handling the HTML code all that well. So new install of xampp and Moodle – some wasted time. Really didn’t like the new Nazi password approach that is now default in the version of Moodle I’m using. More passwords to write down. ;)

Getting outcomes up and going

I’d had outcomes working on the old version of Moodle. My next barrier was getting outcomes to appear on the new. It wasn’t happening simply and I was running out of time, so it sat for a bit. Here’s what I’ve done to get it working:

  • As the Moodle administrator, turn on outcomes under “General Settings”
    Just typing “outcome” in the Moodle adminstrators block was the quickest way to find it.
  • Create some outcomes
    Either in the Admin box under grades or inside an individual course.
  • Possibly add site wide outcomes to the course.
    Outcomes option in course modify box.

Having completed those tasks the theory is that everytime you edit an activity or resource you will have an option to view and select appropriate outcomes.

Outcomes

An outcome has the following data associated with it:

  • Full and short name.
  • Standard outcome – is it available site wide.
  • Scale – which existing scale to use with the outcome.
  • Description – textual description

Outcomes can be imported using as csv file. This could be useful as you could create a set of outcomes for a particular discipline in a CSV file and make them available for anyone to use. Folk at other institutions could import them and have a consistent set of outcomes.

Also, you may not want all discipline outcomes to be available site wide. Could annoy the mathematicians if they kept seeing outcomes from psychology etc. Having outcomes as a CSV would allow these to be imported at a course level. But maybe not…

Checking when outcomes appear

Interested in seeing if the outcomes appear for all activities/resources. Doing a quick test with a couple of courses and reporting where it works. It works for

  • Forums
  • Resource
    • Web page
    • Link to file or web page

Doesn’t work for

  • Labels
    Means of inserting text/HTML into the topics. Used by some to specify readings. Might want to have outcomes on these.
  • Summary

Reflections

As I was doing the above test, a few thoughts arose:

  • What outcomes would you have for a course synopsis?
    For some resources/activities they are too global, too high level to specify a list of outcomes/attributes etc. What do you do with these?

    Given that one of the aims might be to highlight “coverage”, there are some things you wouldn’t allocate anythign to.

  • Why wouldn’t you have outcomes associated with labels?
  • The obvious question which has been bugging me for a while – not all activities/resources for a course are likely to be in the course site. Any curriculum mapping based on the LMS site is not going to be complete. Unless there is some change in practice on the part of the academics. Not a straight forward thing to do.