The following is a summary and some thinking around
Thompson, K., Ashe, D., Carvalho, L., Goodyear, P., Kelly, N., & Parisio, M. (2013). Processing and Visualizing Data in Complex Learning Environments. American Behavioral Scientist, 57(10), 1401–1420. doi:10.1177/0002764213479368
The ability to capture large amounts of data that describe the interactions of learners becomes useful when one has a framework in which to make sense of the processes of learning in complex learning environments. Through the analysis of such data, one is able to understand what is happening in these networks; however, deciding which elements will be of most interest in a specific learning context and how to process, visualize, and analyze large amounts of data requires the use of analytical tools that adequately support the phases of the research process. In this article, we discuss the selection, processing, visualization, and analysis of multiple elements of learning and learning environments and the links between them. We discuss, using the cases of two learning environments, how structure affects the behavior of learners and, in turn, how that behavior has the potential to affect learning. This approach will allow us to suggest possible ways of improving future designs of learning environments.
Some interesting ideas and frameworks/scaffolding for thinking about learning analytics and understanding educational design. Interesting perspective about “big data” techniques not being limited to small amounts of data about lots of people, but also being useful for large amounts of data about small groups of people. Which is linked to the idea of moving analytics beyond use at the macro level into the “micro”.
Remain unsure that some of the work labeled “learning analytics” is what is commonly called learning analytics.
I wonder whether any thought has been given to the application of analytics techniques going beyond the representation/communication stage and extending into action through integration into the educational design? As yet another tool contributing to co-creation and co-configuration.
Very brief intro to learning analytics “focused on making sense of “big data,” data usually collected from learning management systems” used at course and student levels…often to intervene with low/high achieving students…a type of analytics that doesn’t help designers of learning.
educational design defined as “constructing representations of how people should be helped to learn in specific circumstances (Goodyear & Retalis, 2010, p. 10)” and to include the “design of tools, tasks and interactions associated with learning”.
analytics has “mainly focused on the design of courses and analysis on the macro level” and not on “identifying complex patterns of behaviour”. The argument here is to expand “the principels and applications of learning analytics”. Based on “processing and visualizing data in two complex learning environments”
“Networked learning involves people collaborating with the help of technologies in a shared enterprise of knowledge creation” which raises the question for me of “how much ‘help’ do the technologies provide?”.
Focus is exploration of the physical, digital and human elements within learning environments. Objects have properties, intentions and brings values from choices made during design. Thus objects have “effects on human perception and action”.
A distinction of digital technologies is the capacity to change. i.e. it’s protean.
Aside: interesting that the example they give is of customisation of display and not something deeper.
The objects, perception of them and action with them involve various levels of mental/cognitive effort. Additional complication arises from the combination of objects “Thus, only by analyzing the architecture of networks of objects (the pattern of their relations) can one see how design intentions affect what people do, including what they learn. Research of this nature has implica- tions for design work in education” (p. 1403)
Four analytics dimensions
- set design – the physical stage on which learning activity is situated – tools, artifacts etc.
- epistemic design – tasks proposed, knowledge implicated.
- social design – roles, divisions of labor.
- co-creation and co-configuration – since participants’ activities lead to rearrangement of the learning environment.
Aside: it’s reassuring to see explicit mention of change/modification in the last element.
The analytic framework helps identify and represent key elements of complex learning environments – but there’s a need to “develop methods of analysis that incorporate multiple streams of data to describe multiple tool use across multiple tasks” (p. 1403-1404).
The aim is to “reveal process that can inform educational design and student learning processes”. Visualisations of both order and time allow for: “identification of typologies for form” which can help theorize about what works.
After some common quotes about learning analytics, makes the point that rather than “big data” meaning data from lots of people. It can also be “lots of data” about not many people e.g. “short episodes of collaborative work can rapidly create hundreds of gigabytes of data” (p. 1405). This raises difficulties. Expands on tools the developed and considerations.
“We consider learners’ use of the space as important as what they say and the artifacts they create”.
The following figure offers a summary of how patterns are discovered in data. Most definitions of analytics are based on the use of computational methods. The use of human analysis probably doesn’t fit. But it does capture, I think, what actually happens. Certainly part of the data mining activity.
“Finally, the patterns themselves are represented in some way for communication (Figure 1).” (p. 1405)
I have a small problem with the word “finally” in this quote. Not in the context of this paper, but more broadly in learning analytics. At some level representation is enough, but if you do want to make an improvement to educational design, then action is needed. This the argument we make in the IRAC framework – the R is “Representation” i.e. communication. The A is affordance for action.
Now moving onto more specific descriptions of what they’ve done
- focus “on the demonstration of expertise in individual learners as an indicator of successful collaboration”
- Case #1 is an informal networked learning environment (iSpot)
The website (through screenshots) is analysed using semiotics and design to examine the elements of the analytical framework.
- Case #2 – four mater’s students working on a collaborative task.
video data and transcripts analysed to identify indicators of expertise.
Describes iSpot. Mentions earlier analysis (Clow & Makriyannis, 2011).
The focus here is “on the design adopted to make visible a member’s overall level of expertise” – linked somewhat to the outcome of the earlier analysis.
After an introduction to semiotics there is a description of how iSpot represents and calculates the expertise of an individual to illustrate how a design feature is not only a “design element placed in the stage (set design) but in fact encodes a number of underlying meanings, which ultimately reflect a particular way of structuring knowledge (epistemic design) and roles (social design) within the learning network.” (p. 1409).
While this is claimed as drawing on “notions from learning analytics” I’m not sure I see this from the description of what was done. Screenshots of a website followed by semiotic analysis doesn’t quite align with the common definitions of learning analytics I’m familiar with.
Four masters students completing 5 week task. F-t-f meetings captured and analysed. Analytic framework used guide investigation. Automated discourse analysis used examine how learners used the tools, interpreted the task and designed their roles. This group achieved the highest grade in the collaborative component – so looking for identifiable design elements. Description of how this was done.
Through this demonstration, the argument is that the framework has provided added depth to understanding of the co-creation and co-configuration activities in this successful collaboration.
Expands on the options available for further application of learning-analytics techniques, including through the use of a table that draws on components of the figure above as a scaffold.
Learners are influenced by the structure of an environment. The framework here helps identify and theorize about this. Which leads to research/analytics work – at a finer grain. “In so doing, the impacts of design decisions on the behavior of learners can be assessed, and informed redesign work can take place”
I wonder whether the authors are thinking about how environment/tools can be impacted by this “informed redesign”. What if the digital tools capability for modification was informed or even activated by learning analytics? They seem to lean towards this as they finish with
Understanding the relationship between the design of a learning environment and the behavior and learning that occur may enable the design of more effective learning environments.