Here come the indicators, wait for the task corruption

Over the weekend it was reported in various media outlets that the Australian government has a new plan for higher education. A plan that includes words like “results-based funding”, performance, targets, quality and “readily available”. They cynical tone that I hope you are hearing, is not solely – or even mainly – due to some opposition to the idea of universities being accountable or effective. It’s mainly due to the belief that I think for the majority of teaching at universities this move will encourage more task corruption than it will actual real improvement in the quality of teaching.

This post argues for that perception. A bit further down (and in the comments), there is an argument against the practice of requiring new university teaching staff to complete teaching qualifications.

Do you agree? Disagree? Do you know of literature, blog posts or people that have argued otherwise? Are there strategies/advice that can be adopted to limit this sort of stuff (beyond what I’ve suggested at the end?)

Results-based funding: some detail

As outlined in this article there will be four broad areas:

  1. Student participation and inclusion.
  2. Student experience.
  3. Student achievement.
  4. Quality of learning outcomes.

The associated discussion paper goes into more detail.

Section 6 of the discussion paper lists the principles “developed to guide the choice of indicators for performance funding arrangements”. The list includes this one (my emphasis added)

be derived from high quality, objective data sources, and where possible collected at ‘arms length’ by an independent body, as well as not easily manipulated;

Part A of the discussion paper then offers more detail on each of the four measures/indicators. Including the provision of exact information about how the measure will be evaluated. The following information is given on page 14 for the Student Experience measure

  • Goal – improve student engagement and satisfaction.
  • Engagement
    • Target population – 1st year domestic under-graduate students.
    • How target will be articulated – Percentage point improvement in retention rate.
  • See the bit in bold in the above list? Yep, Percentage point improvement in retention rate. i.e. if you pass more first year students, you’ll do better. Retention is also used in the “Student achievement” indicator.

    Obviously pass rate is not a measure that can be “easily manipulated”?

    One perspective on pass rate

    A number of years ago, I was told that one part of one university was seriously considering paying casual teaching staff based on the number of their students who passed. It is my understanding that most people could see the problem with this. If people were paid based on students passing, then most – if not all – of the students would pass. Surely, this is an extreme example?

    Trouble is that it isn’t. Lot’s of evidence of this. The following is taken from Tutty et al (2008)

    More insidiously, there was evidence that rather than just inhibiting change, these
    ‘‘quality measures’’ may actually encourage inferior teaching approaches……The solution to the high failure rate was to change the assessment to satisfy the institutional requirements of satisfied students and reasonable pass rates rather than explore an alternative learning and teaching approach – an effective solution in the current higher education environment that encourages the academic to prioritise other areas, such as research.

    Even if it is not as straight forward as connecting payment to passing, the message in the above is that growing importance of certain quality measures will influence the behaviour of individuals. I believe this is essentially what Goodhart’s law suggestions

    any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

    Oh dear, teaching qualifications again

    It’s with great sadness that I see the following included as a target for the “Quality of Learning Outcomes” indicator

    Agreed increase (n) in proportion of teaching only and teaching and research staff in academic organisation units with a Graduate Certificate in Higher Education or equivalent.

    The obvious assumption here is that if you have a graduate certificate in higher education (or equivalent) you will be a better teacher and your students will have better learning outcomes.

    The trouble with this is two-fold:

    1. The evidence for support of this causal linkage is weak (e.g. Stes et al, 2009).
    2. It ignores the impact of task corruption and Goodhart’s law.

    The first is more difficult to talk about giving perceived limitations in the research. Let’s assume that the connection does exist. What happens when institutions are being encouraged/required to increase the percentage of staff with Graduate Certificates in Higher Education? Goodhart’s law would seem to suggest that the causal connection/statistical regularity will collapse.

    i.e. the staff being forced to complete the Grad Cert will engage in compliance behaviours. They will do enough to get the Grad Cert with the likely outcome that they will not undergo any serious change in their understandings of L&T or in the behaviours. Consequently, student learning outcomes won’t change.

    Better solutions

    This section is included here because some folk complain that I am quick to point the flaws and slow to point to solutions.

    The broadest solution I’ve suggested is given in this presentation. It requires quite a significant mind shift from the current simplistic, quasi-corporate decision making being adopted around learning and teaching to an approach more informed by what we know about complex systems and the motivations and cognition of human beings.

    The simplest solution is embedded in this suggestion on how to improve outcomes on course experience questionnaires. An approach that essentially requires the identification of the smallest changes that can be made to bring courses into line with what we know about student preferences and ensuring that appropriate resources are allocated.

    Another solution I’ve suggested is the REACT process, especially if it were resources the right way. Done correctly, I think it can fulfill one of the main findings from a recent OECD report on teaching quality. i.e.

    Encouraging bottom-up initiatives from the faculty members, setting them in a propitious learning and teaching environment, providing effective support and stimulating reflection on the role of teaching in the learning process all contribute to the quality of teaching.

    References

    Tutty, J., J. Sheard, et al. (2008). “Teaching in the current higher education environment: perceptions of IT academics.” Computer Science Education 18(3): 171-185.

    6 thoughts on “Here come the indicators, wait for the task corruption

    1. I don’t think teaching qualifications are necessarily a bad thing, but it’s a matter of different strokes for different folks. I’d simply prefer more exposure to teaching (outside of what I already do, which isn’t much) and the opportunity to reflect on that.

      I’m not really sure how the GCiTE is structured so I can’t really comment on it. Do you have much first hand experience with it? I’ve yet to hear any real commentary from people who have completed the thing.

      1. By themselves I would agree that teaching qualifications aren’t a bad thing.

        The trouble is that they won’t be by themselves. The current model is that all new staff will be forced to gain such a qualification when they join the institution.

        This may not be a bad thing – I don’t think there’s clear cut evidence that it’s a good thing, but that’s not the argument.

        The argument is what are those new staff going to do when

        1. All the “old hands”, no matter how bad their teaching, will not be required to expend the energy to get a teaching qualification.
        2. The “new hands” will still be under pressure to do research.
        3. At the same time as the teaching qualification they will find that most of the practices/policies that support L&T aren’t conducive to what they are learning about. i.e. it will be difficult to apply what they see.
        4. The rewards process still does not demonstrably value L&T.

        Given this situation, will people engage effectively with the grad cert? Or will they pragmatically complete the requirements so they can tick the box?

        If the latter, then there won’t be any improvement in L&T outcomes for the students as the new academics practice won’t have changed.

    2. Pingback: Teaching Qualifications « t duggan online space

    3. Pingback: Higher Education Teaching Reforms

    4. Pingback: Identifying and designing interventions to improve L&T – a behaviour change framework? « The Weblog of (a) David Jones

    5. Pingback: The realities of the ERA and L&T support services « The Weblog of (a) David Jones

    Leave a Reply

    Please log in using one of these methods to post your comment:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out /  Change )

    Google photo

    You are commenting using your Google account. Log Out /  Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s