Had some time this morning to read Analytics in Higher Education: Benefits, Barriers, Progress and Recommendations from the EDUCAUSE Centre for Applied Research. It’s a report on the results of a survey and seven focus groups (involving IT and Institutional Research – IR – people from universities) attempt to “address the topics of defining analytics, identifying challenges, and proposing solutions” (Bischel, 2012, p. 5). The following uses the quotes from this report that really struck me to identify and suggest what might be the core problem I have with learning analytics. The assumption of rationality.
The report uses this working definition for analytics
Analtyics is the use of data, statistical analysis and explanatory and predictive models to gain insights and act on complex issues
The benefit of analytics
In talking about the benefits of analytics the report says (Bischel, 2012, p. 12)
Others mentioned how a systematic use of anlytics increases the likelihood that faculty, staff, and particularly administrators will base their decisions on data rather than on intuition or conventional wisdom
Ahh, our saviour. Data. The enabler of rationality?
This does seem to be one of the arguments rolled out for moving to learning analytics. Let’s make our decisions based on data, rather than our beliefs.
The problem
I’m currently reading Thinking, Fast and Slow by Daniel Kahneman. The book does a good job of explaining some of the numerous flaws in the decision making capabilities of human beings. A number of the flaws the book covers have to do with how exceedingly poor human beings are at statistical thinking and drawing inferences from data.
It is not reading destined to make you believe that “data-based decision making” is going to save us.
One of the participants in the focus groups seems to make this point
Data are not going to give you a decision. It is your experience and wisdom – that’s what leads you to make decisions. Data are supposed to be a rational approach that provides you a solid foundation for your thinking so that when somebody questions you, you say that ther is a rationale beyhind my data, but still the decision comes from you. The human brain has to make the decision, not an analytical tool. You should have a number of years experience, background and wisdom before you make the decision, but you need ot have the data to thelp you. It’s just a tool, not an end itself.
In the end, it will be people that draw conclusions and make decisions based on what analytics will show them. Human beings that have bounded rationality, competing agendas etc.
But there’s also another problem with this which is summed up by this quote from a blog post from danah boyd
Just because you see traces of data doesn’t mean you always know the intention or cultural logic behind them. And just because you have a big N doesn’t mean that it’s representative or generalizable.
Engage in the complex system
Returning to the EDUCASE working definition
Analtyics is the use of data, statistical analysis and explanatory and predictive models to gain insights and act on complex issues
I don’t like the trend in learning analytics which is reducing it to reports. Reports that are analysed by experts and managers divorced from the environment in which the complex issues arise and on the assumption of rationality used to guide decision making.
Any promise held by analytics would seem to arise from actually using it to engage in the complex issues, observe what happens and learn from that.
Reblogged this on Things I grab, motley collection .
And when you are done with Kahneman you might find Mlodinow, L. (2012). Subliminal : the revolution of the new unconscious and what it teaches us about ourselves (1st ed.). London: Allen Lane a good follow up. This stuff keeps saying to me is that all we are simply apes with a bit of language. Not a lot more. :) Still, the numbers folk will be in the ascendancy for yonks yet. The other good read if you want to mull stats in a somewhat crit light are a couple of little books: Taleb, N. N. (2004). Fooled by Randomness. The Hidden Role of Chance in Life and in the Markets. New York: Random House.
Taleb, N. N. (2007). The Black Swan. The Impact of the Highly Improbable. New York: Random House.
He has a website – smart cookie and makes trouble for the folk who just do the numbers w/o thinking about the assumptions in the analyses they deploy.
Thanks Chris. Have read “The Black Swan” and Kahneman mentions Taleb and the narrative fallacy numerous times. Plan to read more of his stuff.
Hadn’t heard of Mlodinow – watching a talk of his at Google now.
I’d be interested in what you make of Mlodinow. The book is a real hoot. Huge range but beautifully written with a good dash of humour here and there. Again and again, we are reminded of our animal ancestry which, at least in education we seem to have forgotten. The guy did his PhD in theoretical physics. So there may be hope for an old chemist yet. :)
Pingback: The core problem with learning analytics? | E-Learning-Inclusivo (Mashup) | Scoop.it
Pingback: The core problem with learning analytics? « juandon. Innovación y conocimiento
Pingback: The illusion we understand the past fosters overconfidence in our ability to predict the future « The Weblog of (a) David Jones