The following is an attempt to describe early attempts at developing a framework for evaluating existing and developing new interventions aimed at improving learning and teaching within universities. It builds on some ideas from an earlier post.
Sorry, but the first couple of sections seem necessary to convince myself that this is connected with some of my earlier thinking.
Rationale and assumptions
As mentioned many times on this blog, I think most attempts to improve learning and teaching at Universities really don’t work as they don’t change what the majority of academics do. At best they get a bit of compliance. The aim here is to come up with a framework/guidelines that help evaluate and design such interventions.
Empathy driven innovation
While the following uses a framework from psychology, the basic message is the same as a range of other work. That message is, focus on the academics, understand what they are currently doing and experiencing, and design interventions that connect with that experience. Most recently this was discussed in a a post on empathy-driven innovation.
Much of my thinking around improving learning and teaching is based on this sort of approach, the following tries and provide some additional guidance into how to do this. But underpinning it all, is a focus on the actual teaching experience of the academic staff (and the learning experience of the students) as the primary focus.
The aim is to understand what they are experiencing and develop ideas about how that experience can be improved, as perceived by the staff and students and not as perceived by management, the government or some research ideal. The focus is on what the teacher does, not what management does.
Improving L&T is an attempt to change behaviour
An assumption underpinning this thinking is that when you are trying to improve learning and teaching, what you are actually trying to do is to change the behaviour of the teaching academics. If you don’t change the behaviour of the academics, then there is no way you can expect the quality of learning and teaching to change.
This is perhaps the main problem that I have with existing methods being employed to improve learning and teaching at Universities. Existing methods don’t actually consistently result in behaviour change amongst a significant percentage of university teaching staff. I’ve argued before that they don’t work and that there is evidence to support this.
The question is, what’s the solution?
Conceptions and teaching
An example of this mismatch can be seen with the on-going focus on teacher qualifications. For example, the Australian government has introduced KPIs for university learning and teaching that include the use of the % of university teaching staff with teaching qualifications as a KPI. I’ve argued this this attempt will encourage compliance behaviour on the part of institutions and academics.
The institutions are already at it. My current institution is currently introducing a new “desirable criteria” for people in charge of courses – having a formal teaching qualification. The assumption is that this will increase the quality of teaching. It won’t.
At best, a formal teaching qualification will change the conceptions of teaching held by an academic. The “teacher’s thinking” layer in the following diagram. There is significant research (e.g. Trigwell, 2001; Richardson, 2005) that suggests that unless all of the layers in the diagram are in alignment, you won’t get improvement in teaching.
That is, even with a formal teaching qualification, if the context, disciplinary expectations and many other factors aren’t right, then there will be no behaviour change. The teacher will continue to employ the same strategies as always.
How to encourage and enable behaviour change
How to change behaviour is something that the psychologists study. This current line of work is a collaboration between myself and a psychologist colleague. The current aim is to try and distill what is known about behaviour change in psychology into a form that helps explain how someone – with a deep knowledge of the current experience of academics – can encourage and enable behaviour change that results in improved learning and teaching.
In earlier work I’ve drawn on insights from Roger’s (1995) diffusion theory as a guide. There are definite connections between diffusion theory and the following.
The initial spark for this thinking is a paper by Michie et al (2008) which suggests that:
- There are determinants of behaviour change, (see the table below) someone is unlikely to change behaviour when these determinants have the wrong value.
- There are known techniques for encouraging behaviour change.
- These techniques are each seen as likely to influence a certain subset of the determinants.
|Fishbein et al||Michie et al|
|Self-standards||Social/professional role and identity
|Self-efficacy||Beliefs about capabilities|
|Anticipated outcomes/attitude||Beliefs about consequences|
|Intention||Motivation and goals
Memory, attention and decision processes
|Environmental constraints||Environmental context and resources|
From a L&T perspective, the application of this knowledge seems to be something like:
- Identify something that is known to be good practice, that improves L&T, but which isn’t being done in the current context.
- Use the determinants to identify areas of weakness, areas for improvement, around the current context. Especially in terms of the “good” practice.
- Identify generic behaviour change techniques that address those areas of weakness.
- Draw on empathy driven innovation to develop specific interventions, which are based on the information gathered in the previous two steps.
In a previous post that seeks to summarise the Michie et al (2008) paper, I use the following example of the level of interaction or participation of staff in a course site. In the following, I compare/contrast the more traditional approach and the type of approach that I think might work better.
The traditional approach
The more traditional approach is what I suggests is “what management does”. Management will see that there is a problem, i.e. staff aren’t participating enough. They then take it upon themselves to perform some action, which might include:
- Require staff to get a teaching qualification.
The assumption being that with a teaching qualification they know more about the importance of staff interaction and hence will increase their interaction.
- Create a policy that requires a certain level of participation.
At the very least they might say that every course site must have a discussion forum. Over time they might add requirements like: every student query must be answered within two days, or every staff member must make 5 posts a week to the course discussion forum.
- Run a staff development session.
Organise an external expert on the benefits of staff participation to run a session at the institution and invite teaching staff to come along to listen.
- Reward staff who interact at appropriate levels.
Give them extra money or recognition as good teachers. etc.
Most of the above interventions would require a fair bit of work. Most, if not all, of the senior management at the institution would need to be involved, a working party might need to be formed, appropriate consultation, and if they’re really smart they’ll make sure that someone important within the institution is publicly seen encouraging this goal.
I’ve seen the above approach used again and again. I don’t think I’ve ever seen it work to any great extent.
The behavioural change/empathy-driven approach
Using the process above:
- Identify the good practice.
Done, we wish to increase the level of staff participation in a course site/forum.
- Use the determinants to identify areas of weakness in the current context.
As an example, let’s assume that after an appropriate process we have identified the following:
- Skills: many staff don’t have the skills or insight necessary to increase their interaction with their course in effective ways.
- Environmental constraints: there’s nothing in the existing context that helps them perform this task. In fact, the rewards system tends to suggest that research is more important than teaching.
- Norms: there is little visible evidence of what the norms are in terms of appropriate levels of staff participation. Staff don’t know how much other staff are participating, they don’t know what is acceptable and what is not.
- Anticipated outcomes: many staff don’t see the connection between increasing their participation in a course site and benefits to students. They do see how increasing their level of participation increases the time they spend on teaching and decreases the time they spend on research.
- Identify generic behaviour change techniques that target these determinants.
Techniques that target skills include: monitoring; self-monitoring; graded task, starting with easy tasks; modelling/demonstration of behaviour by others; goal/target specified: behaviour or outcome. Norms can be addressed by: modelling/demonstration of behaviour by others; social processes of encouragement, pressure, support; and prompts, triggers, cues. etc.
- Use empathy-driven innovation to design specific interventions.
Embed into the LMS a graph that shows the academic staff members level of participation in their course site (self-monitoring) and compare it against the levels of participation in a group of related courses (social processes of encouragement). Have that graph show up whenever a staff member logs into the LMS. Include in the graph some easily visible representation of the connection between student failure rate and staff participation. Include a link “how to increase participation” to a page that outlines various techniques for increasing participation (prompts, triggers, cues) and includes comments or suggestions from other staff about how they did it (modeling/demonstration of behaviour by others).
Once this intervention is implemented, spend a lot of time observing what happens with the use of this intervention and make on-going change.s
Apart from the obvious, there is one major difference between these two approaches which I must make explicit. The traditional method spends a lot more time on the design of idealistic solutions that are disconnected from the everyday experience (e.g. the setting up and encouraging of people to enrol in a graduate certificate in higher education). Often, such approaches all but ignore the reality of the lived experience (e.g. policies that state a requirement but for which there’s been no work to support implementation). The traditional approach tends to be design heavy.
The empathy-driven approach, focuses on the lived experience, on what happens day-to-day. There is some design work, but much of the time is spent understanding what is going on everyday and trying to respond in an informed way.
It is still very early days for this work. There are lots of questions, here are just some.
How do you evaluate the level of determinants?
In the above example, I outlined some of my ideas of what values the behavioural change determinants around increasing staff participation on a course site might be. Basing the design on my thoughts is not a good start. It has to be based on a deep understanding of the experience/beliefs of the teaching staff.
How can you develop this understanding?
Are there survey instruments or other approaches from psychology that allow you to get some understanding of the current “level” of these determinants across an institution?
Which are the most important determinants?
Michie et al (2008) identify 7 sets of determinants, there are certain to be complexities within each one. In any large population you are likely to get widely differing results.
How do you figure out which of the determinants is most important or difficult?
Answers to this could help you guide the selection of which determinants to address and/or which of the techniques to adopt.
What’s the relationship between the determinants?
It’s likely that there would be relationships between the determinants. For example, if you increase skills in an appropriate way, I would imagine that this would change beliefs about self-efficacy, at least to some level.
What are the known relationships between the determinants?
What are the factors within determinants?
Fishbein et al (2001) talk about norms as being one of the determinants. Michie (et al) 2008 expand this out to include social influences, emotion and action planning. This is just one example, it is likely that each of the determinants embody a collection of factors. For example, self-efficacy might be a combination of characteristics of the person, the type of task and the previous experience of those involved.
What are the factors or sub-components of each of the determinants?
What types of behaviour are there?
Are all behaviours the same? Is a particular teaching behaviour the same as diet or how you control your children?
Are there different categorisations of behaviours with different characteristics?
What are the specifics of change techniques?
Michie et al (2008) just give a short title to these techniques. I imagine that each one is worthy of a literature that seeks to define, describe and evaluate interventions that have been designed on the basis of a specific technique.
What are the specifics of each change technique? What are the success factors? How do you choose which technique might be most appropriate?
Michie, S., Johnston, M., Francis, J., Hardeman, W., & Eccles, M. (2008). From theory to intervention: Mapping theoretically derived behavioural determinants to behaviour change techniques. Applied Psychology: An International Review, 57(4), 660-680.
Rogers, E. (1995). Diffusion of Innovations (4th ed.). New York: The Free Press.