Why Moneyball is the wrong analogy for learning analytics

Learning analytics is one of the research areas I’m interested in. Consequently, I’ve read and listened to a bit about learning analytics over recent times. In that time I’ve often heard Moneyball used as an example or analogy for learning analytics.

I can see the reason for this. It’s a good example of how data can inform decision making in a field many people (especially those in America) are familiar with. Having a best selling book that’s turned into a Brad Pit movie doesn’t hurt either. But I think it’s the wrong analogy for learning analytics.

Moneyball by Kei!, on Flickr
Creative Commons Attribution-Share Alike 2.0 Generic License  by  Kei! 

Why?

As it happens I’ve been reading Nate Silver’s book The Signal and the Noise: Why most predictions fail bus some don’t over recent weeks and I’ll use it to make my case. Silver has had success in applying “analytics” to make predictions in both baseball and US politics and in the book he talks to experts from a range of fields about predictions. Through this process he concluded

I came to realize that prediction in the era of Big Data was not going very well

One of the reasons he gives is

Baseball, for instance, is an exceptional case. It happens to be an especially rich and revealing exception

Why? Well one reason is given when talking about economics, a discipline with a poor track record when it comes to predictions.

This isn’t baseball, where the game is always played by the same rules.

If you don’t play by the rules set down in baseball you are going to get pulled up. What are the rules for learning? How can you be sure that each of the students are aware of the rules, interpreted them the same way, and are playing by them?

A little further on in Silver’s book comes this

The third major challenge for economic forecasters is that their raw data isn’t much good

If the raw data isn’t much good, any predictions you make based on that data is going to have some flaws.

How good is the data in learning? Well, in the face-to-face classroom it’s next to non-existent. At least in the hard, quantitative, consistent form required for most learning analytics. If it’s e-learning, well the data is currently limited to usage logs from the LMS, which are at best a vague indicator of what’s going on.

Intelligent Tutoring Systems tend to solve these problems by having a fixed set of rules (a model) of learning and learners in a particular area. These rules, however, would appear to limit the adoption of the system. How many other contexts can these rules be applied to? Can you actually create such rules for all contexts?

I’m not convinced you can. Especially when broader trends are pushing for an increasingly diverse set of students, but also when learning is seen as a broader, more open and individual happening. Are there “rules” for learning that are broadly applicable?

What the economics analogy suggests for learning analytics

Is learning analytics about prediction? I’d argue that largely it is. You are wanting to understanding what is happening and make predictions about what will happen next. If the learner isn’t going to learn, you want to know that and be able to intervene. You want to make predictions that enable intervention.

The lack of success in prediction in economics suggests that the future of learning analytics may not be bright. At least if it relies on the same models and assumptions as economics. So what needs to change?

About these ads

4 thoughts on “Why Moneyball is the wrong analogy for learning analytics

  1. Pingback: Why Moneyball is the wrong analogy for learning analytics | Analyse This | Scoop.it

  2. suifaijohnmak

    Is learning analytics about predictions? How would you intervene when you found out that the learner is not learning? By sending out emails or commenting on class blog? These sort of strategies have to be integrated with a teacher-machine system to ensure that it is effective. It really depends on what the purpose of LA is. If all the students in your online courses are responding greatly to the discussion boards, you just use the data to show that they are having fruitful interaction. It is the outliers that teachers need to take care of, and perhaps intervene. I wonder if there are such pattern detectors which could do this automatically. John

    Reply
    1. David Jones Post author

      Thanks for picking this up John. The thought about prediction was almost an after thought. Something that came up at the very end of the post when other tasks were calling out for attention.

      So I will try again.

      Some of this confusion comes from a quick use of Silver’s book which is focused on talking about predictions. Hence the use above.

      Is learning analytics about predictions? I’m thinking yes.

      SoLAR defines learning analytics as

      the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.

      Increasingly I am thinking that that “optimizing” part is the crux of it. i.e. the aim of learning analytics, at least for a teacher/student, is to make a change in their learning – to optimize it. This suggests that the insight gained from analysing the data is being used to plan some intervention, some change. Embedded in that change is the assumption that it will get better/different. To me this is a type of prediction.

      Without this type of prediction, I’m not sure learning analytics is all that useful. Except perhaps to researchers and archeologists.

      In your example, you do get confirmation that there are some that are working okay, but you’re still keen to intervene for the outliers.

      In terms of automatic pattern detectors, if there aren’t someone already, I’m sure there are some folk working on it.

  3. suifaijohnmak

    Hi David, Yes, that is resonating: the aim of learning analytics, for a teacher/student, is to make a change in their learning – to optimize it. In the case of a connectivist course, I do see it makes sense, as there are feedback loops whilst interactions in the forums, discussion boards, and so the instructors would be able to make adjustment when he/she realize the difficulties that learners might have. Rita, Fournier and I have used SNA in analysing the how facilitators’ participation had impacted on the number of postings and comments by MOOCs participants, and we found that facilitator’s involvement did impact on participants’ engagement and discussion. I just wonder how this could be applicable with an instructivist MOOC with tens or hundreds thousands of participants (like the xMOOCs) where discussion boards are “closed”. May be this is where I would like to explore, if LA be applicable to xMOOCs and how instructors would interpret the LA.

    Reply

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s