Designing for situation awareness in a complex system

The following is a summary and probably some thoughts on

Endsley, M. (2001). Designing for Situation Awareness in Complex System. In Proceedings of the Second International Workshop on the symbiosis of humans, artifacts and environment. Kyoto, Japan.

@beerc is excited by some of the potential of this and related work for university e-learning. It seems to fit with our thoughts that most universities and the individuals therein aren’t even scratching the surface in terms of what technology could offer.

My initial thoughts

I like some of the initial outlining of the problem, however, I think the solution smacks too much of the complicated systems approach, rather than the complex adaptive approach. The solution is essentially a specialised waterfall model (requirements/design/evaluate) with a focus on situation awareness. There’s some interesting stuff there, but I’m not sure how applicable it is to university e-learning. I remain leery of this idea that experts can come in an analyse the problem and fix it. It needs to be more evolutionary.

There are some nice quotes for higher ed and its systems.

The challenge of the information age

“The problem is no longer lack of information, but finding what is needed when it is needed.” Actually, I’d have to argue that when it comes to information about the learners in the courses I teach, I’m still suffering the former problem when it should be the latter.

Describes “The information gap” where “more data != more information”. Draws on some of the common explanations.

From data to information

Draws on a Bennis (1977) quote “This post-technological age has been defined as one in which only those who have the right information, the strategic knowledge, and the handy facts can make it”….makes the point “making the right decisions will depend on having a good grasp of the true picture of the situation”.

The overflow of data needs translating into information. But it will “need to be processed and interpreted slightly differently by different individuals, each of whom has varied dynamically changing but inter-related information needs”.

This translation “depends on understanding how peopel process and utilize information in the decision making activities”

Understanding “human error”

A few examples of this before “the biggest challenge within most industries and the most likely cause of an accident receives teh label of human error. This is a most misleading term, however, that has done much to sweep the real problems under the rug.”

Instead it’s argued that the human was “striving against significant challenges”….coping “with hugely demanding complex systems”. Overload in terms of data and technology. This is addressed through long lists of procedures and checklists which are apt to eventually fail. Instead

The human being is not the cause of these errors, but the final dumping ground for the inherent problems and difficulties in the technologiges we have created. The operator is usually the one who must bring it all together and overcome whatever failures and inefficiencies exist in the system

This resonates quite strongly with my experience at different universities when trying to teach a large course with the provided information systems.

Situation awareness: The key to providing information

“Developing and maintaining a high level of situation awareness is the most difficult part of many jobs”. SA is defined as “an internalised mental model of the current state of the operator’s environment…..This integrated picture forms the central organising feature from which all decision making and action takes place”.

Developing and keeping SA up to date makes up a vast portion of the person’s job.

“The key to coping in the ‘information age’ is developing systems that support this process. This is where our current technologies have left human operators the most vulnerable to error.”…..cites research that shows SA problems were “the leading causal factor”.

I wonder if such research could be done in a contemporary university setting?

Success will come is you can combine and present the vast amounts of data in a way that provides SA. “The key here is in understanding that true situation awareness only exists in the mind of the human operator”.

This is an interesting point given the rush to automated analytics.

The successful improvement of SA through design or training problems requires the guidance of a clear understanding of SA requirements in the domain, the individual, system and environmental factors that affect SA, and a design process that specifically addresses SA in a systematic fashion

SA defined

Citing Endsley (1988) SA is defined as

the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future”

and there a levels of SA

  1. Perception of the elements in the environment

    “perceiving the status, attributes and dynamics of relevant elements in the environment”

  2. Comprehension of the current situation

    More than awareness of the elements, includes “an understanding of the significance of those elements in light of one’s goals”.

    A novice operator may achieve the same Level 1 SA as an experienced operator, but will likely fall short at Level 2.

  3. Projection of future status

    What are the elements in the environment going to do?

Theoretical underpinnings

Links to broader literature that has developed a theoretical framework model. Apparently heavily based on the cognitivist/psychology research. Working memory, long term memory etc. e.g. Fracker’s (1987) hypothesis that working memory is the main bottleneck for situational awareness and other perspectives. Mental models/schema get a mention as a solution.

“Of prime importance is that this process can be almost instantaneous due to the superior abilities of human pattern matching mechanisms”. Hence the importance of expertise and experience.

Designing for situation awareness enhancement

The type of systems integration required for SA

usually requires very unique combinations of information and portrayals of information that go far beyond the black box “technology oriented” approaches of the past

Designing these systems is complex, but progress made. Too complex to cover here, but talks about three major steps

  1. SA requirements analysis

    Frequently done with a form of cognitive task analysis/goal-directed task analysis. The point is that goals/objectives form the focus, NOT tasks.

    Done using a combination of cognitive engineering procedures with a number of operators.

    Done (with references) in many domains.

  2. SA-Oriented design

    Presents 6 design principles for SA, that is also applicable more broadly.

  3. Measurement of SA in design evaluation

    Mentions the Situation Awareness Global Assessment Technique (SAGAT) measuring operator SA.

When initially reading those three steps my first reaction was “Arggh, it’s the SDLC/waterfall model all over again. That’s extremely disappointing”. I then started wondering if this was because they are thinking of complicated systems, not complex adaptive systems?

Staff need to be using the same tools they use to teach to also learn

The title of this post is from a presentation by someone at a University responsible for the institutional e-learning systems. It doesn’t matter which university because I imagine it’s a line that has been used at quite a few of them. It does matter that I think it’s completely wrong-headed and illustrates perfectly the problem with institutional e-learning systems and the processes and people that support them.

They are designed to ensure people use the provided systems, rather than what’s best for learning.

philosophy by erix!, on Flickr
Creative Commons Attribution 2.0 Generic License  by  erix! 

They’ll be better at the LMS if we use the LMS to support them

The idea is that any staff development that occurs should be done via the LMS and other institutional e-learning systems. The benefit of this is that learning through these tools not only addresses a learning need, but it also provides teachers with experience from the perspective student.

Who learns with an LMS?

What would happen if I ran a survey asking people what tools they use to learn every day?

I’d imagine tools like Google, Twitter, Diigo, Pinterest etc would be near the top. I don’t imagine an LMS would be anywhere near the top.

It’s a focus on the selected tool (hammer), not on learning (the egg)

The problem with this statement

Staff need to be using the same tools they use to teach to also learn

is that it reflects the mindset that what’s best for learning is using the tools that have already been adopted by the institution. Those tools are the starting point.

What’s not the starting point are the tools people are already using, or the tools that are better for learning. Especially for the time when they stop studying at the institution. This connects to my recent post about the failure of institutional eportfolios.

Another example is getting help with Moodle. Moodle is the LMS used by the institution for which I work. When I want to learn about something related to Moodle I use Google which invariably takes me to either the main Moodle site or some of the good quality Moodle related resources shared on the websites of other institutions (e.g. UNSW). It is my understand that I will never find any of the Moodle how-to resources created by my current institution because they reside in a Moodle instance that isn’t searchable by Google. An example of how the focus is on the tool, not on how people actually learn.

Another example is past experience when talking about BIM. BIM is essentially a tool to enable the use of individual student blogs. But whenever central L&T folk at a Moodle institution hear blogs, their first question is something like, “Did you know that Moodle has blogs built-in?”.

If all you have is a hammer….

BIM and BAD

This post arises from two events today

  1. The ASCILITE’2014 call for papers came out today and I’m thinking about a paper I might submit.
  2. The first #edc3100 assignment is due today and my use of BIM has struck a unique problem that I need to solve.

The third is that I’m a touch fried from answering queries about “I submitted my assignment the wrong way” (the main problem with these queries is that they mean I have to engage with a horrible online assignment submission system) that I need to engage in something else.

The paper

The working title for the paper I’m thinking of is “The story of BIM: Being BAD as a way to bridge rhetoric and reality”.

BAD is an acronym that captures what I think is missing from the institutional approach to university e-learning

  1. Bricolage – the LMS as Enteprise Systems doesn’t allow or cater for bricolage.
  2. Affordances – resulting in an inability to leverage the affordances of technology to improve learning and teaching.
  3. Distribution – the idea that knowledge about how to improve L&T is distributed and the implications that has for the institutional practice of e-learning.

    i.e. current methods rely on the single, unified view of learning and teaching. A view that is expressed most concretely in the form of the LMS.

    This component will draw on a range of related “network” type theoretical perspectives including connectivism, Complex Adaptive Systems, embodied cognition and ANT – to name but a few.

The idea is if institutional e-learning is to get better, it needs to be BAD (more BAD?).

The following is an example of how the reality of using BIM in action supports the idea that it needs to be BAD. Or at least it’s a very small step. It captures the messiness (the distribution) of e-learning in a typical university course. A messiness that isn’t captured properly by PRINCE II and other methodologies, hierarchical organisational structures, appropriately total quality assured forms and processes and “theory” based abstractions like adopter categories.

And yes, there is some strong connection (repetition) with earlier perspectives/frameworks of mine. If at first you don’t succeed, try, try again.

Background

300+ students in EDC3100 are currently using their own blogs to reflect on their learning journey (to varying levels of engagement). Their blog posts contribute almost 15% of their final result in the course.

BIM is being used to keep a track of what they are sharing. The students create their blogs on WordPress.com (or anywhere they like) and register them with BIM. BIM then keeps a copy of all their posts.

But BIM’s capability don’t match the learning design I’m using in this course. BIM was originally designed to have student posts made in response to specific prompts/questions and then have the posts marked manually be human markers. In EDC3100, the students blog about anything they want. We offer some prompts, but they can ignore them. Student posts aren’t marked, instead students have to post a certain number of posts (on average) every week, the posts have to average a certain word count, and a certain number have to contain links to online resources and the blog posts of other students.

This analysis of student posts and the subsequent mark they get is done by a program I wrote. A bit of bricolage that takes bits and pieces of information extracted (with some difficulty) from various institutional systems and makes use of them in a way that solves my problem.

With a bit of bricolage each of the 300+ students have received an email recently telling them what the system knows about their progress. This gives them sometime to tick all the boxes prior to the first assignment being due (yep, still have due dates, haven’t travelled too far from the well-worn paths).

The problem

One student has reported a problem with what the system knows about their blog. The system says that only one of the posts links to another student post, but the student’s blog actually has two posts that link to other student posts. This is confirmed.

But no-one else is reporting the problem. There’s something unique about this student’s blog that has picked up a bug in the system.

The uniqueness of this bug appears to me as one of the problems associated with the failure of institutional systems to deal with the Distribution aspect of BAD. In a complex, distributed knowledge network there is no one view. But the trad approach can only ever respond to one view. This argument needs a bit of work.

Typically this problem is because the author of the post has used a link other to the other student’s blog. The program I wrote knows the URLs for all the student blogs. It checks all the links in a post against the known student posts.

I’ve visually checked the students blog posts in BIM and they are showing valid links to student blogs.

Argghh.

The solution – Chrome is too smart – it’s distribution

This is what I see when I look at the blog post using the Chrome browser

Chrome by David T Jones, on Flickr

The link that is shown is to the blog of another student. The program should pick this up and count it as a link.

Here’s what I see when I view it under the Firefox browser

Firefox by David T Jones, on Flickr

See the difference?

The student appears to have used some form of URL shortener. Looks like a WordPress tool. While this shortened URL does point to the post of another student. My little system doesn’t know how to convert a shortened URL into a full URL. So it doesn’t count it.

It appears that I must have a plugin installed on Chrome (or perhaps Chrome is smart enough on its own) to automatically expand out the wp.me shortened URL into the full link and change what is shown to the user.

I as the user is ignorant of this change happening.

Not a bad example of Distribution. How cognition/smarts/learning is distributed amongst all of the tools. Change on bit of the network and the outcome changes.

Some areas of improvement for #edc3100

It’s over a month since my last post. Start of semester will do that to you. 350+ students having EDC3100, ICT and Pedagogy inflicted upon them always makes for a busy time. No more so with the first assignment due in a couple of days. In a slight turn up for the books, rather than complain about the holes in the various institutional systems and processes make this worse, it’s time to save a couple of observations for future consideration.

The following is written as if it is the fault or limitations of students that are the problem here. It’s not intended that way. The course contributes, as does many other factors.

Knowing how to solve a problem

A feature/problem with the course is it doesn’t focus on teaching specific ICTs. Students are expected to use Diigo, WordPress and perhaps Twitter as part of their learning and beyond pointing to some online resources there is limited instruction in using these tools. There are a couple of screencasts showing some common tasks and explaining how it fits in the course.

But there will be problems, and it appears that a fair number of students aren’t very good at recognising the need to employ the tech cheat sheet, let alone use it.

Activities early on need to help develop this.

Accept resources at face value

Students are shown Scootle and love it. Especially the integration with the Australian Curriculum. Select your content descriptor, look at the elaboration and click on the Scootle link to see a list of resources that match. Learning design made easy!?

But many don’t see (or haven’t explicitly mentioned where I can see) some of the reservations I have about these repositories and the quality of the information in them.

Enabling them to be more evaluative of the quality of the resources would seem to be a good idea.

Aren’t focusing on learning yet

The first assignment is asking them to come up with the three reasons for using ICTs to enhance student learning. So far, many of the reasons are of the general “enable 21st century learning” fluffy-bunny type observation, rather than because of specific ways in which the use of ICTs will improve the learning experience/outcome.

Given the material and experiences they have this early in the course, that’s not surprising. Fluffy bunnies is about all they’ve seen. Not sure it gets significantly better throughout the course. Need to work on that.

Not showing off the new knowledge society

One of the objectives of EDC3100 is

Examine, discuss and reflect on core ideas about the knowledge generation and knowledge economy and the implications of these for curriculum and pedagogy

As part of this we show the obligatory “the world has changed because of ICTs” type videos, but then we proceed to return to largely traditional learning and teaching practices. The course has moved on a bit, but I’m not sure it’s enough. Especially in terms of assessment.

One of the students did pick up on this point in one of her blog posts. Of course, I didn’t save that post.

Need to do better at this.

A story about the failure of institutional eportfolios

In which I relate a personal story about how the one eportfolio I was required as a student to make on an institutional eportfolio system has now disappeared for good (to me) with no communication from the institution.

I’m a long-term skeptic when it comes to institutionally chosen eportfolio systems like Mahara, PebblePad etc. Back in January 2009 I expressed my first disquiet with eportfolios in terms of how institutions approach innovation around e-learning. i.e. Ohh, everyone is installing eportfolios, let’s leap on that bandwagon and expends lots of resources encouraging/requiring everyone to use this fad while we ignore all the contextual opportunities and issues within the institution. This is a cycle you can see repeated with new values for eportfolio: open source LMS, learning analytics, OERs, MOOCs….

I’ve also wondered just how long institutions were going to continue supporting the eportfolios created by their students. I know have an answer for one institution, about 3 years.

2011 – Reluctant eportfolio author

Given my skepticism about eportfolios it was somewhat ironic that I was required as a student to create an eportfolio when studying to become a high school teacher. My fellow students and I were all required to use Mahara to create an eportfolio showing off evidence that we had met each of the relevant teacher standards. This was a requirement as part of the course and came with the expectation that we’d use it in interviews.

2012 – Ongoing use

As it happened, I didn’t become a school teacher and didn’t use my eportfolio in interviews.

Instead I got a job at another university teaching a course on ICT and Pedagogy. The course design I had to use in the first year of teaching that course required the students to create their assignments in the new institution’s instance of Mahara. Mahara’s not the easiest of tools to learn, so I used my existing eportfolio as an example in the course.

It appears that I wasn’t the only one. My alma mater also appeared to be using the eportfolio I created as an example. This is based on the following request from one of their students

Since you’re the guru of the e.portfolio I was hoping that you wouldn’t mind telling me how to create the little tabs at the top of each page as you have in yours.

2013 – Declining use

Given the hassles in using Mahara for student assignments, I changed the course to remove the use of Mahara. Instead students were creating their own blog on their choice of service. Didn’t point my students to my eportfolio but still got the odd evidence of on-going use at the other institution.

2014 – It’s gone!

This year the program I teach into is trying to encourage students to continually think about gathering evidence against the Australian Professional Standards for Teachers (APST) and preparing their eportfolio. We’re also encouraged to link our course activities to the APST.

Being the good corporate citizen that I am, I was modifying my course site to add these reminders in. It seemed a good opportunity to point to my eportfolio as an example of what is required. Where was that link? Ahh, there it is? Oops!!

Missing eportfolio by David T Jones, on Flickr

It’s gone! The entire Mahara site no longer exists. How could that be?

It appears that what was once an externally hosted site is now hosted internal to the organisation.

What’s more the eportfolios on that site cannot be accessed unless you go through the institution’s version of Moodle. My student account – which was working for the old system – will no longer give me access via this new method.

Would’ve been nice to be told

I understand the need to re-evaluate how services are provided, but it would’ve been nice to be told that the system was moving so that I could have made use of the eportfolio communities much vaunted standards for sharing content. A dump of the site and move to a new one would make sense.

Perhaps this is something I should have done ages ago. However, I also think there’s an argument to be made that the institution could of informed ex-students of the move. After all, the institution’s Alumni association still communicates regularly with me.

If institution’s provide an institutional eportfolio system, doesn’t this imply a larger burden of support? Moving beyond the forget about the student once they graduate approach to maintaining a life-long relationship?
I’m sure I read something like that in some marketing spiel.

If that additional burden isn’t going to be picked up, then isn’t there at least the expectation that you will clearly communicate what the service level will be? After all, SLAs are the beloved of all central IT departments.

If this burden isn’t being picked up, then what does this mean for institutions that are requiring students to create eportfolios for use in interviews and other post-enrolment requirements?

Evernote as a “solution” to a #moodle “problem”

One of the problems I have with the way Moodle has been implemented at this institution (and probably not unique to this institution) is the absence of a search engine. Mostly because the students “inability to find anything” is being blamed on poor course design and being solved by an institutional push for consistency.

Amongst the many problems with this “solution” (e.g. the assumption that a single design will be suitable for the full spectrum of courses, topics and pedagogies used across a university) is that it won’t actually solve the “inability to find anything” for my course. My course will still have a series of activities and resources in which useful information is stored. No amount of consistent layout of a Moodle site is going to make it simple to find specific bits of information amongst those activities/resources. You need a search engine.

The purpose of this post is to explore a solution to this problem and also to illustrate the use of the TEST Framework. The TEST framework is introduced in the first week of my course as one tool to help the students think about the analysis of different tools to solve a task.

Task

To able to find specific bits of information on the EDC3100 course website (called a “Study Desk”) and help those taking the course to also find those bits of information they need.

Environment

  • The course website is hosted by an instance of Moodle.
  • There’s no search engine for the site and access to the site is restricted.
  • A Mac laptop is my main computing device with some use of an iPhone and the occasional use of an institutional PC running Windoze.

Skills

I’m a competent, experienced computer user with some pretentions toward software development.

Tools

  1. Run my own search engine?

    Tools like regain or Arch potentially offer the ability to search the entire EDC3100 site using my credentials.

    On the downside, it would really only be of use for EDC3100. The installation and setup cost could be large as both are open source projects at a stable stage of their life cycle. It’s also not an option that other EDC3100 folk could follow, nor would it easily allow me to provide access to those other folk.

  2. Save the content.

    Most of the material is made available as Moodle books that can be printed out and through this saved as PDF files on my laptop. This would allow use of the Mac’s local search facility.

    It’s all a bit manual, but low-tech and would only work for EDC3100. i.e. a specific task for EDC3100. It also isn’t an option that would help the other people in the course to be able to find information.

  3. Diigo and social bookmarking.

    We are using Diigo for social bookmarking in the course. Bookmarking particular topics and being able to search through those bookmarks is one option.

    Problem is it assumes you know what you’ll want to find later when you read it/bookmark it. Doubt it would scale real well. Especially with lots of people bookmarking in a group. On the other hand, if it is restricted to my bookmarks, there’s the problem that the words I use mean nothing (or something completely different) to the other staff and students.

  4. Evernote or other note-taking software.

    There is a category of software to support notetaking. Evernote is one of the more prevalent examples, especially in education. Wikipedia has a comparison of Notetaking software.

    A tool like Evernote provides clients across a range of platforms – allowing use on different devices. It supports some form of search and the ability to organise what is being saved. Evernote is being used by some as a research tool. Teachers are also using it.

    With that last example use by teachers, there is a shared notebook that can be searched by people other than the creator. Even does a bit of OCR-enabled searching on images. I could potentially create a shared EDC3100 notebook that others can search.

    But the real point here is the ability to demonstrate something that students could do for themselves. i.e. keep their own notes on the course in ways that make sense to them.

  5. Put all the content on an open website.

    This was the option I initially considered and rejected because of the additional workload (I never did get around to doing it last year).

    To some extent the Evernote option is an example of this approach. The main difference is that the “open website” provides a range of clients across different devices that I can use to place information on the site.

Getting started with Evernote

  1. Download the Mac client for Evernote and install it.
  2. Create a free account.
  3. Use the Getting started with Evernote guide.
  4. Create a note, add an image.
  5. Install the Evernote web clipper
  6. Trial the web clipper on the EDC3100 website.

    The article clipper auto-detects the main content of a page.

    The Moodle book print option opens up a browser window without the widgets, but copy and paste that URL into a normal browser window and all is good. That looks like it might work.

  7. Set up an EDC3100 notebook and make it public
  8. Add some of the information.
  9. Do a search on the public notebook. All good.

Some reflection and experimentation

This does mean that I’ll need to update the notebook as I modify the resources in the study desk. But it’s a very easy process, so should be okay.

One limit will be that I won’t make the discussion forum content available here for privacy reasons, but on the plus side, Moodle does offer a search engine for that.

I also need to see if there’s any simple way to present the information in a more useful way. At the moment the public notebook just lists the resources added in reverse chronological order. Ahh, there are some view options at the bottom, but only simply sorts by title, age etc. I can tag the resources and have started using tags to indicate the week of content.

Another slide downside is that the search facility only really displays the notes in which the search term appears and then highlights it in the note. It doesn’t actually provide any support for you to go specifically to the place in the note where the search term was found. It appears from some early searching, that public notebooks are not indexed by Google. That could have helped.

That’s enough to try this out and see what folks think.

An ateleological quote

Came across this presentation, “Learning how to learn” (a bit ra-ra, but covers many of the bases for changes in the world and implications for learning). It includes this quote

If you focus on results, you will never change. If you focus on change, you will get results — Jack Dixon

The quote is used widely, but I can’t find the original citation.

Ateleological connection

This quote resonates with my view that there needs to be more ateleological processes around organisations – especially schools and universities – and less of the traditional teleological processes.

A teleological process focuses on results. It defines what the goal is and measures progress against movement toward that goal. If you aren’t helping achieve that goal you’re wrong. Such an approach is a poor match for the current context of most organisations. It assumes that your existing schemata will allow you to think of what the future needs, that what will be required in the future won’t change before you get there, and that you can actually get a large group of people (e.g. academics) to work together for the same goal. Focus on the results, and you will never change.

An ateleological process focuses on the next small change to make within the very local context and focuses on making those changes fast and continuous whilst keeping in synch with what is going on in the world and the organisation. Focus on the change and you will get results.