The core problem with learning analytics?

Had some time this morning to read Analytics in Higher Education: Benefits, Barriers, Progress and Recommendations from the EDUCAUSE Centre for Applied Research. It’s a report on the results of a survey and seven focus groups (involving IT and Institutional Research – IR – people from universities) attempt to “address the topics of defining analytics, identifying challenges, and proposing solutions” (Bischel, 2012, p. 5). The following uses the quotes from this report that really struck me to identify and suggest what might be the core problem I have with learning analytics. The assumption of rationality.

The report uses this working definition for analytics

Analtyics is the use of data, statistical analysis and explanatory and predictive models to gain insights and act on complex issues

The benefit of analytics

In talking about the benefits of analytics the report says (Bischel, 2012, p. 12)

Others mentioned how a systematic use of anlytics increases the likelihood that faculty, staff, and particularly administrators will base their decisions on data rather than on intuition or conventional wisdom

Ahh, our saviour. Data. The enabler of rationality?

This does seem to be one of the arguments rolled out for moving to learning analytics. Let’s make our decisions based on data, rather than our beliefs.

The problem

I’m currently reading Thinking, Fast and Slow by Daniel Kahneman. The book does a good job of explaining some of the numerous flaws in the decision making capabilities of human beings. A number of the flaws the book covers have to do with how exceedingly poor human beings are at statistical thinking and drawing inferences from data.

It is not reading destined to make you believe that “data-based decision making” is going to save us.

One of the participants in the focus groups seems to make this point

Data are not going to give you a decision. It is your experience and wisdom – that’s what leads you to make decisions. Data are supposed to be a rational approach that provides you a solid foundation for your thinking so that when somebody questions you, you say that ther is a rationale beyhind my data, but still the decision comes from you. The human brain has to make the decision, not an analytical tool. You should have a number of years experience, background and wisdom before you make the decision, but you need ot have the data to thelp you. It’s just a tool, not an end itself.

In the end, it will be people that draw conclusions and make decisions based on what analytics will show them. Human beings that have bounded rationality, competing agendas etc.

But there’s also another problem with this which is summed up by this quote from a blog post from danah boyd

Just because you see traces of data doesn’t mean you always know the intention or cultural logic behind them. And just because you have a big N doesn’t mean that it’s representative or generalizable.

Engage in the complex system

Returning to the EDUCASE working definition

Analtyics is the use of data, statistical analysis and explanatory and predictive models to gain insights and act on complex issues

I don’t like the trend in learning analytics which is reducing it to reports. Reports that are analysed by experts and managers divorced from the environment in which the complex issues arise and on the assumption of rationality used to guide decision making.

Any promise held by analytics would seem to arise from actually using it to engage in the complex issues, observe what happens and learn from that.

7 thoughts on “The core problem with learning analytics?

  1. And when you are done with Kahneman you might find Mlodinow, L. (2012). Subliminal : the revolution of the new unconscious and what it teaches us about ourselves (1st ed.). London: Allen Lane a good follow up. This stuff keeps saying to me is that all we are simply apes with a bit of language. Not a lot more. :) Still, the numbers folk will be in the ascendancy for yonks yet. The other good read if you want to mull stats in a somewhat crit light are a couple of little books: Taleb, N. N. (2004). Fooled by Randomness. The Hidden Role of Chance in Life and in the Markets. New York: Random House.
    Taleb, N. N. (2007). The Black Swan. The Impact of the Highly Improbable. New York: Random House.

    He has a website – smart cookie and makes trouble for the folk who just do the numbers w/o thinking about the assumptions in the analyses they deploy.

    1. I’d be interested in what you make of Mlodinow. The book is a real hoot. Huge range but beautifully written with a good dash of humour here and there. Again and again, we are reminded of our animal ancestry which, at least in education we seem to have forgotten. The guy did his PhD in theoretical physics. So there may be hope for an old chemist yet. :)

  2. Pingback: The core problem with learning analytics? | E-Learning-Inclusivo (Mashup) |

  3. Pingback: The core problem with learning analytics? « juandon. Innovación y conocimiento

  4. Pingback: The illusion we understand the past fosters overconfidence in our ability to predict the future « The Weblog of (a) David Jones

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s