Designing for situation awareness in a complex system

The following is a summary and probably some thoughts on

Endsley, M. (2001). Designing for Situation Awareness in Complex System. In Proceedings of the Second International Workshop on the symbiosis of humans, artifacts and environment. Kyoto, Japan.

@beerc is excited by some of the potential of this and related work for university e-learning. It seems to fit with our thoughts that most universities and the individuals therein aren’t even scratching the surface in terms of what technology could offer.

My initial thoughts

I like some of the initial outlining of the problem, however, I think the solution smacks too much of the complicated systems approach, rather than the complex adaptive approach. The solution is essentially a specialised waterfall model (requirements/design/evaluate) with a focus on situation awareness. There’s some interesting stuff there, but I’m not sure how applicable it is to university e-learning. I remain leery of this idea that experts can come in an analyse the problem and fix it. It needs to be more evolutionary.

There are some nice quotes for higher ed and its systems.

The challenge of the information age

“The problem is no longer lack of information, but finding what is needed when it is needed.” Actually, I’d have to argue that when it comes to information about the learners in the courses I teach, I’m still suffering the former problem when it should be the latter.

Describes “The information gap” where “more data != more information”. Draws on some of the common explanations.

From data to information

Draws on a Bennis (1977) quote “This post-technological age has been defined as one in which only those who have the right information, the strategic knowledge, and the handy facts can make it”….makes the point “making the right decisions will depend on having a good grasp of the true picture of the situation”.

The overflow of data needs translating into information. But it will “need to be processed and interpreted slightly differently by different individuals, each of whom has varied dynamically changing but inter-related information needs”.

This translation “depends on understanding how peopel process and utilize information in the decision making activities”

Understanding “human error”

A few examples of this before “the biggest challenge within most industries and the most likely cause of an accident receives teh label of human error. This is a most misleading term, however, that has done much to sweep the real problems under the rug.”

Instead it’s argued that the human was “striving against significant challenges”….coping “with hugely demanding complex systems”. Overload in terms of data and technology. This is addressed through long lists of procedures and checklists which are apt to eventually fail. Instead

The human being is not the cause of these errors, but the final dumping ground for the inherent problems and difficulties in the technologiges we have created. The operator is usually the one who must bring it all together and overcome whatever failures and inefficiencies exist in the system

This resonates quite strongly with my experience at different universities when trying to teach a large course with the provided information systems.

Situation awareness: The key to providing information

“Developing and maintaining a high level of situation awareness is the most difficult part of many jobs”. SA is defined as “an internalised mental model of the current state of the operator’s environment…..This integrated picture forms the central organising feature from which all decision making and action takes place”.

Developing and keeping SA up to date makes up a vast portion of the person’s job.

“The key to coping in the ‘information age’ is developing systems that support this process. This is where our current technologies have left human operators the most vulnerable to error.”…..cites research that shows SA problems were “the leading causal factor”.

I wonder if such research could be done in a contemporary university setting?

Success will come is you can combine and present the vast amounts of data in a way that provides SA. “The key here is in understanding that true situation awareness only exists in the mind of the human operator”.

This is an interesting point given the rush to automated analytics.

The successful improvement of SA through design or training problems requires the guidance of a clear understanding of SA requirements in the domain, the individual, system and environmental factors that affect SA, and a design process that specifically addresses SA in a systematic fashion

SA defined

Citing Endsley (1988) SA is defined as

the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future”

and there a levels of SA

  1. Perception of the elements in the environment

    “perceiving the status, attributes and dynamics of relevant elements in the environment”

  2. Comprehension of the current situation

    More than awareness of the elements, includes “an understanding of the significance of those elements in light of one’s goals”.

    A novice operator may achieve the same Level 1 SA as an experienced operator, but will likely fall short at Level 2.

  3. Projection of future status

    What are the elements in the environment going to do?

Theoretical underpinnings

Links to broader literature that has developed a theoretical framework model. Apparently heavily based on the cognitivist/psychology research. Working memory, long term memory etc. e.g. Fracker’s (1987) hypothesis that working memory is the main bottleneck for situational awareness and other perspectives. Mental models/schema get a mention as a solution.

“Of prime importance is that this process can be almost instantaneous due to the superior abilities of human pattern matching mechanisms”. Hence the importance of expertise and experience.

Designing for situation awareness enhancement

The type of systems integration required for SA

usually requires very unique combinations of information and portrayals of information that go far beyond the black box “technology oriented” approaches of the past

Designing these systems is complex, but progress made. Too complex to cover here, but talks about three major steps

  1. SA requirements analysis

    Frequently done with a form of cognitive task analysis/goal-directed task analysis. The point is that goals/objectives form the focus, NOT tasks.

    Done using a combination of cognitive engineering procedures with a number of operators.

    Done (with references) in many domains.

  2. SA-Oriented design

    Presents 6 design principles for SA, that is also applicable more broadly.

  3. Measurement of SA in design evaluation

    Mentions the Situation Awareness Global Assessment Technique (SAGAT) measuring operator SA.

When initially reading those three steps my first reaction was “Arggh, it’s the SDLC/waterfall model all over again. That’s extremely disappointing”. I then started wondering if this was because they are thinking of complicated systems, not complex adaptive systems?

Staff need to be using the same tools they use to teach to also learn

The title of this post is from a presentation by someone at a University responsible for the institutional e-learning systems. It doesn’t matter which university because I imagine it’s a line that has been used at quite a few of them. It does matter that I think it’s completely wrong-headed and illustrates perfectly the problem with institutional e-learning systems and the processes and people that support them.

They are designed to ensure people use the provided systems, rather than what’s best for learning.

philosophy by erix!, on Flickr
Creative Commons Attribution 2.0 Generic License  by  erix! 

They’ll be better at the LMS if we use the LMS to support them

The idea is that any staff development that occurs should be done via the LMS and other institutional e-learning systems. The benefit of this is that learning through these tools not only addresses a learning need, but it also provides teachers with experience from the perspective student.

Who learns with an LMS?

What would happen if I ran a survey asking people what tools they use to learn every day?

I’d imagine tools like Google, Twitter, Diigo, Pinterest etc would be near the top. I don’t imagine an LMS would be anywhere near the top.

It’s a focus on the selected tool (hammer), not on learning (the egg)

The problem with this statement

Staff need to be using the same tools they use to teach to also learn

is that it reflects the mindset that what’s best for learning is using the tools that have already been adopted by the institution. Those tools are the starting point.

What’s not the starting point are the tools people are already using, or the tools that are better for learning. Especially for the time when they stop studying at the institution. This connects to my recent post about the failure of institutional eportfolios.

Another example is getting help with Moodle. Moodle is the LMS used by the institution for which I work. When I want to learn about something related to Moodle I use Google which invariably takes me to either the main Moodle site or some of the good quality Moodle related resources shared on the websites of other institutions (e.g. UNSW). It is my understand that I will never find any of the Moodle how-to resources created by my current institution because they reside in a Moodle instance that isn’t searchable by Google. An example of how the focus is on the tool, not on how people actually learn.

Another example is past experience when talking about BIM. BIM is essentially a tool to enable the use of individual student blogs. But whenever central L&T folk at a Moodle institution hear blogs, their first question is something like, “Did you know that Moodle has blogs built-in?”.

If all you have is a hammer….