Daily Archives: January 18, 2011

Analytics creating too much transparency? A two-edged sword?

Have been listening to a Dave Snowden podcast of a “101 organic KM course”. Amongst many familiar themes is the mention of the pitfalls of too much transparency hurting innovation.

He uses the example of expense accounts to illustrate the point. At one stage he had a large expense account which could be used to fund interesting and unusual approaches around his work. The innovation was possible was there was no itemisation/justification of the expense. Upon moving into a large company there came a requirement for itemisation. That itemisation kills off innovation.

This rings a bell at the moment, because of the current discussion about the problems with learning analytics and in particular George Siemens’ list of concerns.

In the dim dark past of the 90s, when I was an innovative, young university academic no-one took any notice of what I did within the courses I was teaching. I could do a lot of very different things that are documented in my publications from that time. Not all of them worked as I planned, but they all helped something interesting grow.

In part this was possible because of the very problem that often worried me about some of my colleagues. At that time, there were at least 2 or 3 of my fellow academics who were fairly widely known as being really bad educators. Even though one or two claimed to be great teachers, even a cursory glance at their practice and resources or a chat with a range of their students would confirm some really, really bad practice. What annoyed me at the time was that the system allowed their practice to be opaque. As long as they met various deadlines (even though they were often late) and had a reasonable grade distribution there practice was allowed to continue.

What I am only now starting to realise is that if that system wasn’t opaque, if it were too transparent, I probably wouldn’t have undertaken any of the innovative work I did. One explanation why not arises from Siemens’ list of concerns. In a university with analytics baked in and heavily relied upon by management to “manage”

  • The act of providing a quality learning experience has been reduced to a set of numbers and graphs that specify certain activities and tasks. In response to known patterns from analytics I am expected to perform certain tasks, perhaps even push certain buttons at certain times to encourage those patterns to happen again.
  • What is accepted is what is measured and has become the target. Anyone moving away from the established pattern is fighting the inertia of the organisation and its systems. (This was actually one of the problems I faced working within an institution with a history of industrial print-based education in the mid-1990s while attempting to use the Internet).
  • Different interpretations of what is good learning/teaching due to the diversity inherent in the disciplines, concepts, individual students and teachers is lost. You (and the students) are expected to follow the standard patterns that analytics has established as effective. (This is also my problem with the LMS. For some institutions it has become the case that you can do any online learning you want. As long as the functionality is provided within the LMS restrictions of quiz, discussion forum, assignment management etc.)
  • The smart/pragmatic academics and students will have identified what “analytics patterns” are required and figured out the least painful way to provide those requirements.
  • When something like the recent Queensland floods occur it will throw the analytics system into melt-down as the expect patterns won’t be there. For example, the two “late” letters I received in the post today (first post since before Christmas due to the floods) from Video Ezy asking for their DVD back. Regardless of floods cutting off all possibility of me returning it.
  • The “analytics patterns” will drive management to change policy and funding for practices so that only those patterns can be re-created. Anything that falls outside that norm will not be funded. (e.g. this is one of the major, unsolved problems the industrial, print-based distance education university had with online, it kept funding for f-t-f and DE, never figuring out that online could be different).
  • Since the “analytics patterns” have been established and the funding routinised management are able to treat the folk responsible for designing and delivering teaching like building blocks that can be replaced as needed.

And there’s more.

Learning analytics looks like being a two-edged sword.