Using the PIRAC – Thinking about an “integrated dashboard”

On Monday I’m off to a rather large meeting to talk about what data might be usefully syndicated into a integrated dashboard. The following is an attempt to think out lod about the (P)IRAC framework (Jones, Beer and Clark, 2013) in the context of this local project. To help prepare me for the meeting, but also to ponder some recent thoughts about the framework.

This is still a work in progress.

Get the negativity out of the way first

Dashboards sux!!

I have a long-term negative view of the value of dashboards and traditional data warehouses/business intelligence type systems. A view that has risen out of both experience and research. For example, the following is a slide from this invited presentation. There’s also a a paper (Beer, Jones, & Tickner, 2014) that evolved from that presentation.

Slide19

I don’t have a problem with the technology. Data warehouse tools do have a range of functionality that is useful. However, in terms of providing something useful to the everyday life of teachers in a way that enhances learning and teaching, they leave a lot to be desired.

The first problem is the Law of Instrument.

Hammer ... Nail ... by Theen ..., on Flickr
Creative Commons Creative Commons Attribution-Noncommercial-Share Alike 2.0 Generic License   by  Theen … 

The only “analytics” tool the institution has is the data warehouse, so that’s what it has to use. The problem being is that the data warehouse cannot be easily and effectively integrated into the daily act of learning and teaching in a way that provides significant additional affordances (more on affordances below).

Hence it doesn’t get used.

Now, leaving that aside.

(P)IRAC

After a few years of doing learning analytics stuff, we put together the IRAC framework as an attempt to guide learning analytics projects. Broaden the outlook and what needed to be considered. Especially what needed to be considered to ensure that the project outcome was widely and effectively used. The idea is that the four elements of the framework could help ponder what was available and what might be required. The four original components of IRAC are summarised in the following table.

IRAC Framework (adapted from Jones et al 2013)
Component Description
Information
  • the information we collect is usually about “those things that are easiest to identify and count or measure” but which may have “little or no connection with those factors of greatest importance” (Norman, 1993, p. 13).
  • Verhulst’s observation (cited in Bollier & Firestone, 2010) that “big data is driven more by storage capabilities than by superior ways to ascertain useful knowledge” (p. 14).
  • Is the information required technically and ethically available for use?
  • How is the information to be cleaned, analysed and manipulated?
  • Is the information sufficient to fulfill the needs of the task?
  • In particular, does the information captured provide a reasonable basis upon which to “contribute to the understanding of student learning in a complex social context such as higher education” (Lodge & Lewis, 2012, p. 563)?
Representation
  • A bad representation will turn a problem into a reflective challenge, while an appropriate representation can transform the same problem into a simple, straightforward task (Norman, 1993).
  • To maintain performance, it is necessary for people to be “able to learn, use, and reference necessary information within a single context and without breaks in the natural flow of performing their jobs.” (Villachica et al., 2006, p. 540).
  • Olmos and Corrin (2012) suggest that there is a need to better understand how visualisations of complex information can be used to aid analysis.
  • Considerations here focus on how easy is it to understand the implications and limitations of the findings provided by learning analytics? (and much, much more)
Affordances
  • A poorly designed or constructed artefact can greatly hinder its use (Norman, 1993).
  • To have a positive impact on individual performance an IT tool must be utilised and be a good fit for the task it supports (Goodhue & Thompson, 1995).
  • Human beings tend to use objects in “ways suggested by the most salient perceived affordances, not in ways that are difficult to discover” (Norman, 1993, p. 106).
  • The nature of such affordances are not inherent to the artefact, but are instead co-determined by the properties of the artefact in relation to the properties of the individual, including the goals of that individual (Young, Barab, & Garrett, 2000).
  • Glassey (1998) observes that through the provision of “the wrong end-user tools and failing to engage and enable end users” even the best implemented data warehouses “sit abandoned” (p. 62).
  • The consideration for affordances is whether or not the tool and the surrounding environment provide support for action that is appropriate to the context, the individuals and the task.
Change
  • Evolutionary development has been central to the theory of decision support systems (DSS) since its inception in the early 1970s (Arnott & Pervan, 2005).
  • Rather than being implemented in linear or parallel, development occurs through continuous action cycles involving significant user participation (Arnott & Pervan, 2005).
  • Buckingham-Shum (2012) identifies the risk that research and development based on data already being gathered will tend to perpetuate the existing dominant approaches from which the data was generated.
  • Bollier and Firestone (2010) observe that once “people know there is an automated system in place, they may deliberately try to game it” (p. 6).
  • Universities are complex systems (Beer, Jones, & Clark, 2012) requiring reflective and adaptive approaches that seek to identify and respond to emergent behaviour in order to stimulate increased interaction and communication (Boustani et al., 2010).
  • Potential considerations here include, who is able to implement change? Which, if any, of the three prior questions can be changed? How radical can those changes be? Is a diversity of change possible?

Adding purpose

Whilst on holiday enjoying the Queenstown view below and various refreshments, @beerc and I discussed a range of issues, including the IRAC framework and what might be missing. Both @beerc and @damoclarky have identified potential elements to be added, but I’ve always been reluctant. However, one of the common themes underpinning much of the discussion of learning analytics at ASCILITE’2014 was for whom was learning analytics being done? We raised this question somewhat in our paper when we suggested that much of learning analytics (and educational technology) is mostly done to academics (and students). Typically in the service of some purpose serving the needs of senior management or central services. But the issue was also raised by many others.

Which got us thinking about Purpose.

Queenstown View

As originally framed (Jones et al, 2013)

The IRAC framework is intended to be applied with a particular context and a particular task in mind……Olmos & Corrin (2012), amongst others, reinforce the importance for learning analytics to start with “a clear understanding of the questions to be answered” (p. 47) or the task to be achieved.

If you start the design of a learning analytics tool/intervention without a clear idea of the task (and its context) in mind, then it’s going to be difficult to implement.

In our discussions in NZ, I’d actually forgotten about this focus in the original paper. This perhaps reinforces the need for IRAC to become PIRAC. To explicitly make purpose the initial consideration.

Beyond increasing focus on the task, purpose also brings in the broader organisational, personal, and political considerations that are inherent in this type of work.

So perhaps purpose encapsulates

  1. Why are we doing this? What’s the purpose?
    Reading between the lines, this particular project seems to be driven more by the availability of the tool and a person with the expertise to do stuff with the tool. The creation of a dashboard seems the strongest reason given.
    Tied in with seems to be the point that the institution needs to be seen to be responding to the “learning analytics” fad (the FOMO problem). Related to this will, no doubt, be some idea that by doing something in this area, learning and teaching will improve.
  2. What’s the actual task we’re trying to support?
    In terms of a specific L&T task, nothing is mentioned.
  3. Who is involved? Who are they? etc.
    The apparent assumption is that it is teaching staff. The integrated dashboard will be used by staff to improve teaching?

Personally, I’ve found thinking about these different perspectives useful. Wonder if anyone else will?

(P)IRAC analysis for the integrated dashboard project

What follows is a more concerted effort to use PIRAC to think about the project. Mainly to see if I can come up with some useful questions/contributions for Monday.

Purpose

  • Purpose
    As above the purpose appears to be to use the data warehouse.

    Questions:

    • What’s the actual BI/data warehouse application(s)?
    • What’s the usage of the BI/data warehouse at the moment?
    • What’s it used for?
    • What is the difference in purpose in using the BI/data warehouse tool versus Moodle analytics plugins or standard Moodle reports?
  • Task
    Without knowing what the tool can do I’m left with pondering what information related tasks that are currently frustrating or limited. A list might include

    1. Knowing who my students are, where they are, what they are studying, what they’ve studied and when the add/drop the course (in a way that I can leverage).
      Which is part of what I’m doing here.
    2. Having access to the results of course evaluation surveys in a form that I can analyse (e.g. with NVivo).
    3. How do I identify students who are not engaging, struggling, not learning, doing fantastic and intervene?

    Questions:

    • Can the “dashboards” help with the tasks above?
    • What are the tasks that a dashboard can help with that isn’t available in the Moodle reports?
  • Who
  • Context

What might be some potential sources for a task?

  1. Existing practice
    e.g. what are staff currently using in terms of Moodle reports and is that good/bad/indifferent?

  2. Widespread problems?
    What are the problems faced by teaching staff?
  3. Specific pedagogical goals?
  4. Espoused institutional priorities?
    Personalised learning appears to be one. What are others?

Questions:

  • How are staff using existing Moodle reports and analytics plugins?
  • How are they using the BI tools?
  • What are widespread problems facing teaching staff?
  • What is important to the institution?

Information

The simple questions

  • What information is technically available?
    It appears that the data warehouse includes data on

    • enrolment load
      Apparently aimed more at trends, but can do semester numbers.
    • Completion of courses and programs.
    • Recruitment and admission
      The description of what’s included in this isn’t clear.
    • Student evaluation and surveys
      Appears to include institutional and external evaluation results. Could be useful.

    As I view the dashboards, I do find myself asking questions (fairly unimportant ones) related to the data that is available, rather than the data that is important.

    Questions

    • Does the data warehouse/BI system know who’s teaching what when?
    • When/what information is accessible from Moodle, Mahara and other teaching systems?
    • Can the BI system enrolment load information drill down to course and cohort levels?
    • What type of information is included in the recruitment and admission data that might be useful to teaching staff?
    • Can we get access to course evaluation surveys for courses in a flexible format?
  • What information is ethically available?

Given the absence of a specific task, it would appear

Representation

  • What types of representation are available?
    It would appear that the dashboards etc are being implemented with PerformancePoint hence it’s integration with Sharepoint (off to a strong start there). I assume relying on its “dashboards” feature hence meaning it can do this. So there would appear to be a requirement for Silverlight to see some of the representations

    Questions

    • Can the data warehouse provide flexible/primitive access to data?
      i.e. CSV, text or direct database connections?
  • What is knowledge is required to view those representations?
    There doesn’t appear to be much in the way of contextual help with the existing dashboards. You have to know what the labels/terminology mean. Which may not be a problem for the people for whom the existing dashboards are intended.
  • What is the process for viewing these representations?

Affordances

Based on the information above about the tool, it would appear that there are no real affordances that the dashboard system can provide. It will tend to be limited to representing information.

  • What functionality does the tool allow people to do?
  • What knowledge and other resources are required to effectively use that functionality?

Change

  • Who, how, how regularly and with what cost can the
    1. Purpose;
      Will need to be approved via whatever governance process exists.
    2. Information;
      This would be fairly constrained. I can’t see much of the above information changing. At least not in terms of getting access to more or different data. The question about ethics could potentially meant that there would be less information available.
    3. Representation; and,
      Essentially this would appear that all the dashboards can change. Any change will be limited by the specifics of the tool
    4. Affordances.
      You can’t change what you don’t have.

    be changed?

Adding some learning process analytics to EDC3100

In Jones and Clark (2014) we drew on Damien’s (Clark) development of the Moodle Activity Viewer (MAV) as an example of how bricolage, affordances and distribution (the BAD mindset) can add some value to institutional e-learning. My empirical contribution to that paper was talking about how I’d extended MAV so that when I was answering a student query in a discussion forum I could quickly see relevant information about that student (e.g. their major, which education system they would likely be teaching into etc).

A major point of that exercise was that it was very difficult to actually get access to that data at all. Let alone get access to that data within the online learning environment for the course. At least if I had to wait upon the institutional systems and processes to lumber into action.

As this post evolved, it’s become also an early test to see if the IRAC framework can offer some guidance in designing the extension of this tool by adding some learning process analytics. The result of this post

  1. Defines learning process analytics.
  2. Applies that definition to my course.
  3. Uses the IRAC framework to show off the current mockup of the tool and think about what other features might be added.

Very keen to hear some suggestions on the last point.

At this stage, the tool is working but only the student details are being displayed. The rest of the tool is simply showing the static mockup. This afternoon’s task is to start implementing the learning process analytics functionality.

Some ad hoc questions/reflections that arise from this post

  1. How is the idea of learning process analytics going to be influenced by the inherent tension between the tendency for e-learning systems to be generic and the incredible diversity of learning designs?
  2. Can generic learning process analytics tools help learners and teachers understand what’s going on in widely different learning designs?
  3. How can you the diversity of learning designs (and contexts) be supported by learning process analytics?
  4. Can a bottom-up approach work better than a top-down?
  5. Do I have any chance of convincing the institution that they should provide me with
    1. Appropriate access to the Moodle and Peoplesoft database; and,
    2. A server on which to install and modify software?

Learning process analytics

The following outlines the planning and implementation of the extension of that tool through the addition of process analytics. Schneider et al (2012) (a new reference I’ve just stumbled across) define learning process analytics

as a collection of methods that allow teachers
and learners to understand what is going on in a learning scenario, i.e. what participants work(ed) on, how they interact(ed), what they produced(ed), what tools they use(ed), in which physical and virtual location, etc. (p. 1632)

and a bit later on learning scenario and learning process analytics are defined as

as the measurement and collection of learner actions and learner productions, organized to provide feedback to learners, groups of learners and teachers during a teaching/learning situation. (p. 1632)

This is a nice definition in terms of what I want to achieve. My specific aim is to

collect, measure, organise and display learner actions and learner productions to provide feedback to the teacher during a teaching/learning situation

Two main reasons for the focus on providing this information to the teacher

  1. I don’t have the resources or the technology (yet) to easily provide this information to the learners.
    The method I’m using here relies on servers and databases residing on my computer (a laptop). Not something I can scale to the students in my class. I could perhaps look at using an external server (the institution doesn’t provide servers) but that would be a little difficult (I haven’t done it before) and potentially get me in trouble with the institution (not worth the hassle just yet).

    As it stands, I won’t even be able to provide this information to the other staff teaching into my course.

  2. It’s easier to see how I can (will?) use this information to improve my teaching and hopefully student learning.
    It’s harder to see how/if learners might use any sort of information to improve their learning.

Providing this information to me is the low hanging fruit. If it works, then I can perhaps reach for the fruit higher up.

Learner actions and productions

What are the learner actions and productions I’m going to generate analytics from?

The current course design means that students will be

  1. Using and completing a range of activities and resources contained on the course site and organised into weekly learning paths.
    These actions are in turn illustrated through a range of data including

    • Raw clicks around the course site stored in system logs.
    • Activity completion.
      i.e. if a student has viewed all pages in a resource, completed a quiz, or posted the required contributions to a discussion forum they are counted as completing an activity. Students get marks for completing activities.
    • Data specific to each activity.
      i.e. the content of the posts they contributed to a forum, the answers they gave on a quiz.
  2. Posting to their individual blog (external to institutional systems) for the course.
    Students get marks for # of posts, average word count and links to other students and external resources.
  3. Completing assignments.
  4. Contributing to discussions on various forms of social media.
    Some officially associated with the course (e.g. Diigo and others unofficially (student Facebook groups).

I can’t use some of the above as I do not have access to the data. Private student Facebook groups is one example, but the more prevalent is institutional data that I’m unable to access. In fact, the only data I can easily get access to is

  • Student blog posts; and,
  • Activity completion data.

So that’s what I’ll focus on. Obviously there is a danger here that what I can measure (or in this case access) is what becomes important. On the plus side, the design of this course does place significant importance on the learning activities students undertake and the blog posts. It appears what I can measure is actually important.

Here’s where I’m thinking that the IRAC framework can scaffold the design of what I’m doing.

Information

Is all the relevant Information and only the relevant information available?

Two broad sources of information

  1. Blog posts.
    I’ll be running a duplicate version of the BIM module in a Moodle install running on my laptop. BIM will keep a mirror of all the posts students make to their blogs. The information in the database will include

    • Date, time, link and total for each post.
    • A copy of the HTML for the post.
    • The total number of posts made so far, the url for the blog its feed.
  2. Activity completion.
    I’ll have to set up a manual process for importing activity completion data into a database on my computer. For each activity I will have access to the date and time when the student completed the activity (if they have).

What type of analysis or manipulation can I perform on this information?

At the moment, not a lot. I don’t have a development environment that will allow me to run lots of complex algorithms over this data. This will have to evolve over time. What do I want to be able to do initially? An early incomplete list of some questions

  1. When was the last time the student posted to their blog?
  2. How many blog posts have they contributed? What were they titled? What is the link to those posts?
  3. Are the blog posts spread out over time?
  4. Who are the other students they’ve linked to?
  5. What activities have they completed? How long ago?
  6. Does it appear they’ve engaged in a bit of task corruption in completing the activities?
    e.g. is there a sequence of activities that were completed very quickly?

Representation

Does the representation of the information aid the task being undertaken?

The task here is basically giving me some information about the student progress.

For now it’s going to be a simple extension to the approach talked about in the paper. i.e. whenever my browser sees on a course website a a link to a user profile, it will add a link [Details] next to it. If I click on that link I see a popup showing information about that student. The following is a mockup (click on the images to see a larger version) of what is currently partially working

001 - Personal Details

By default the student details are shown. There are two other tabs, one for activity completion and one for blog posts.

Requirement suggestion: Add into the title of each tab some initial information. e.g. Activity completion should include something like “(55%)” indicating the percentage of activities currently completed. Or perhaps it might be the percentage of the current week’s activities that have been completed (or perhaps the current module).

The activity completion tab is currently the most complicated and the ugliest. Moving the mouse of the Activity Completion tab brings up the following.

002 - Activity completion

The red, green and yellow colours are ugly and are intended to indicate a simple traffic light representation. Green means all complete, red is not, yellow means in progress for some scale.

The course is actually broken up into 3 modules. The image above shows each module being represented. Open up a module and you see the list of weeks for that module – also with the traffic light colours. Click on a particular week and you see the list of activities for that week. Also with colours, but also with the date when the student completed the activity.

Requirement suggestion: The title bars for the weeks and modules could show the first and last time the student completed an activity in that week/module.

Requirement suggestion: The date/time when an activity was completed could be a roll-over. Move the mouse over the date/time and it will change the date/time to how long ago that was.

Requirement suggestion: What about showing the percentage of students who have completed activities? Each activity could show the % of students who had completed it. Each week could show the percentage of students who had completed that week’s activities. Each module could….

Requirement suggestion: Find some better colours.

The blog post tab is the most under-developed. The mockup currently only shows some raw data that is used to generate the students mark.

003- blog posts

Update The following screen shot shows progress on this tab. The following is from the working tool.

BlogProcessAnalytics

Requirement suggestions:

  • Show a list of recent blog post titles that are also links to those posts.
    Knowing what the student has (or hasn’t) blogged recently may give some insight into their experience.
    Done: see above image.
  • Show the names of students where this student has linked to their blog posts.
  • Organise the statistics into Modules and show the interim mark they’d get.
    This would be of immediate interest to the students.

Affordances

Are there appropriate Affordances for action?

What functionality can this tool provide to me that will help?

Initially it may simply be the display of the information. I’ll be left to my own devices to do something with it.

Have to admit to being unable to think of anything useful, just yet.

Change

How will the information, representation and the affordances be Changed?

Some quick answers

  1. ATM, I’m the only one using this tool and it’s all running from my laptop. Hence no worry about impact on others if I make changes to what the tool does. Allows some rapid experimentation.
  2. Convincing the organisation to provide an API or some other form of access directly (and safely/appropriately) to the Moodle database would be the biggest/easiest way to change the information.
  3. Exploring additional algorithms that could reveal new insights and affordances is also a good source.
  4. Currently the design of the tool and its environment is quite kludgy. Some decent design could make this particularly flexible.
    e.g. simply having the server return JSON data rather than HTML and having some capacity on the client side to format that data could enable some experimentation and change.

References

Schneider, D. K., Class, B., Benetos, K., Lange, M., Internet, R., Developer, A., & Zealand, N. (2012). Requirements for learning scenario and learning process analytics. In World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1632–1641).

A proposal for fixing what’s broken with ed tech support in some universities

This paper analyses the outcomes of what a small group of academics (myself included) attempted to do to develop the knowledge/capability to develop effective learning for hundreds of pre-service teachers via e-learning. That experience is analysed using a distributive view of knowledge and learning and illustrates just how broken what passes for ed tech support/academic staff development in some universities. Picking up on yesterday’s post, the paper reports on academics harnessing their digital fluency to address the almost complete lack of usefulness of the institutionally developed attempts at supporting academic staff in developing the knowledge necessary for effective e-learning.

The distributive view of knowledge and learning used in the paper drew on three conceptual themes from Putnam and Borko (2000) and one theme we’ve added. Those themes suggests that knowledge and learning is/should be

  1. situated;
    Context matters. An inappropriate context can limit transfer of learning into different contexts. The entire system/context in which learning takes place is fundamental to what is learned.
  2. social;
    How we think and what we know arises from on-going interactions with groups of people of time.
  3. distributed; and,
    The knowledge required to perform a task does not exist solely within an individual person or even groups of people. It also resides in artifacts. Appropriate tools can enhance, transform and distribute cognition and expand “a system’s capacity for innovation and invention” (Putnam & Borko, 2000, p. 10).
  4. protean.
    The computer is “the first metamedium, and as such has degrees of freedom and expression never before encountered” (Kay, 1984, p. 59) it has a “protean nature”. i.e. digital technology can be flexible and should be open to be manipulated in response to needs.

This post is an attempt to propose one way in which institutional attempts at ed tech support could be transformed to actually support these four themes. i.e. to actually be made useful and appropriate for the task.

Analysing existing practice

When I first started on this post the plan was to analyse my existing institution’s attempts at ed tech support. So I logged into the Moodle site for the course I’ll be teaching next semester and asked the question, “If I had problem with X (whatever that is), how would I find an answer?”. The answer to that question was summarised in this post. A post that is password protected because of the embarrassing difficulty I had in answering that question.

Using the four themes the following criticisms might be made

  1. situated;
    The support resources were not situated in the context. I could not find any help with the course site from within the course site. I had to go to another website and waste time figuring out which labyrinth of links I would follow to get to the support resources. The first time I tried it I failed.

    I just wanted to check some aspect of my earlier analysis. Initially I had difficulties finding my way through the labyrinth.

  2. social;
    Almost the entire support resources were centrally produced and approved. Some with heavy production values. Pure information distribution. There was a small collection of Moodle discussion forums intended I imagine to encourage social interaction. Half of those forums had no posts, the other half had single posts all from the same author.

    Apart from the discussion forums, the only way to add to these resources was via the small number of people from central support.

    This also means that the “message” shared via these resources can be controlled by the institution. Raising the question about whether differing views can be expressed. For example, there is a section on using the Mahara e-portfolio that extols the educational virtues of Mahara. There’s no way I can contribute the reasons I don’t use Mahara and use something different. The point isn’t that Mahara is a bad tool, but that there are some issues with using it and alternatives. More importantly, there’s no way to share this alternate view.

  3. distributed; and
    Any content to be added to the site had to be manually added by a small select group of people. There was no integration between the resources and other systems. For example, the IT Help Desk system was in no way integrated. So if there was a known problem with the “discussion forums” being raised through the IT help desk, there was no way for that information to appear in the support resources on “discussion forums”.
  4. protean.
    The support resources were implemented in an ad hoc collection of Moodle-based course sites. The resources were all static and professionally designed. Little or no way to repurpose those resources or to add to them.

    Moodle discussion forums can generate RSS feeds and also have the option of subscribing to a forum via email. These are methods that allow a user to modify how they interact with the discussion forum. If I’m interested in a forum I can integrate new activity on the forum into my daily routine either through my feed reader or email.

    The ability to grab the RSS feed or subscribe via email to the discussion forums in these support resources are not visible via the interface that has been used.

Some design principles

Beyond critiquing what exists, the four conceptual themes above might also be useful in terms of developing guidelines for what might be. Here’s an initial brainstorm of potential guidelines. Feel free to add and argue.

  1. The support resources should be situated within the context of the academics.
    Some of what this might suggests includes

    • If you want to learn about using the discussion forums better, you should be able to do this from within the discussion forum.
    • If you want to know how to use the discussion forum for a introductory/warm up activity at the start of semester, this should be possible.
    • The support you receive should be tailored(able) to the type of course or discipline you are teaching.
    • The support system should know who you are, what you’re teaching, what you’ve done before, what groups you belong to, what time of semester (e.g. before semester starts, first couple of weeks, assessment due, end of semester etc) it is etc.
    • The support system should use the tools that people use (not the institution).
      i.e. not this from Dutton

      Organizations aren’t thinking about the ‘networked individual’ – the networking choices and patterns of individual Internet users. They’re still focused on their own organizational information systems and traditional institutional networks.

  2. The support area should encourage/enable participation in various discourse communities.
    Which might suggest approaches such as

    • The system should make you aware of the communities/individuals that are using the tool you are currently (thinking of) using.
      e.g. if you’re looking at using the BIM Moodle module the support system should help you become aware of who else has used the tool and perhaps how (leading into..)
    • The system should help capture and make available for on-going use the “residue of experience” (Riel & Pollin, 2004) of other members of the community.
      The discussion, reflections and analysis of prior use of tools and methods should be available. At a simple level, this might be ensuring that any and all questions about the discussion forum (including those from the helpdesk) be visible/searchable from the support site about the discussion forum.
  3. The support area should integrate with and integrate into it all of the appropriate organisational and external systems and processes.
    This might include such things as

    • Knowledge from other systems offering support appear automatically.
      For example, any known issues information about tools are integrated appropriately into the environment.
    • Organisational information sources such as student records systems, teaching responsibilities databases, results of course evaluation surveys etc should be integrated into the support and used to situate and modify resources appropriately.
    • Knowledge from the support area should be openly available (as appropriate) for integration into other systems.
      Might be as simple as generating an RSS/OPML feed (or two) or allowing email subscription. Perhaps publish an API.
    • The “how to do” advice in the support area should actually help you do it.
      i.e. rather than a sequence of steps describing what you do, there’s actually a link that will take you back to actual system and help you do it. Linked to the idea of Context Appropriate Scaffolding Assemblages (CASA).
  4. The support area should support manipulation and change by the users and their actions (protean).
    This might mean

    • Something as simple as having decent customisation options.
    • Something more radical like Smallest Federated Wiki.
      i.e. where each individual or group could fork the support resources and make their own changes. Changes that might be potentially integrated back into the original institutional version.

One illustration

So how might that work in action. Here’s one possible illustration.

Login

You start by logging in to one of the institutional systems (e.g. the LMS).

Straight away I have a qualm about whether or not a login is required. In order for the system to know about you (see the situated principle above) some form of identification is required. But requiring a login means that system isn’t open. So perhaps there’s an avenue that doesn’t require a login.

The Mini-map appears

Not only do you login to the LMS but you also login to the “support system” and the mini-map appears.

The mini-map is a small icon (or three) that appears in the browser. Perhaps in the top right hand corner of the page. From now on, where ever you go the mini-map is there. But as you move around to different systems it will likely change because it knows your situation and responds accordingly.

This is based on the mini-maps concept from games occuring in immersive 3D worlds. The suggestion isn’t that this mini-map be represented as an actual map (though perhaps it might be), the point is that the purpose is to help orient you within the e-learning space.

What the mini-map might do

Nesbitt et al (2009) suggest that a

mini-map might also display the position of key landmarks along with the position of the player’s avatar and any other relevant actors in the game

which gives some idea of what the “mini-map” in this context might do.

Specific functionality might include

  1. You are here.
    Provide a summary of what it knows about your current location within the teaching and learning environment. This might include insight into the time of term, common or required tasks you may need to complete soon (or have completed at similar times in the past), updates and announcements updating you on what’s going on in the environment since you were last here.

    e.g. new problems that have arisen around where you are. Such as the lecture capture system being down and that it is being worked upon. This would also suggest that the support system is independent (distributed) from the various services. So it can keep working if they are down.

  2. Who else is here.
    Let you know who else is on this particular page, or who else is using this particular service in another course or at another time. e.g. other people who have used this particular service. Provide some functionality to allow you to control and organise who you want to know about.
  3. What have they done
    Access to the residue of experience, what have these people actually done within your current location. What worked. What didn’t. This might also be links to literature etc.
  4. How to do stuff.
    Advice on how to perform various tasks. Pedagogical patterns, learning designs etc including potential CASA’s that would help or do stuff for you.

And on a non-institutional system

The mini-map would appear when you visit any online location that has been used for learning and teaching. For example, if you want to Google Drive you would have access to (almost) the same functionality described above.

If the mini-map didn’t appear, because you’ve visited a tool that no-one else has used before, you could choose to add the tool to the mini-map and that addition would then be visible to others.

Implementation

I see the mini-map being implemented with something like a Greasemonkey script. This is how it’s possible for it to appear independent of whether you’re viewing an institutional or non-institutional system.

It might work something like the following

  1. You’ve installed the Greasemonkey script on your browser.
  2. You can choose to enable or disable the script at anytime.
  3. Then, whenever you visit a web page the mini-map grabs the URL for that page and sends it to a server.
  4. The server checks to see if that URL matches anything supported by the mini-map.
  5. The version of the mini-map that is displayed depends on whether the URL is currently supported
    • Supported – then show the full mini-map.
    • Not supported – then show the minimal mini-map with just the option to “add this page”
  6. If viewing the supported mini-map you then have access to a range of functionality.
  7. Some functionality will pop up new information.
    e.g. click on the “People” icon and the mini-map might show a list of people you “know” that are/have been here.
  8. Some functionality will take you into a different system.
    e.g. click on one of the people in the list and you might get taken to a web page that shows what they were doing, when, and also provides access to details of what others have done.

The different systems used to provide the various support services should tend to be whatever makes sense but with a focus on being tools people use all the time and not limited to institutional tools. You might use Slack for some functions. SFW might be good for others.

An interesting and challenging extension to this would be to allow the “mini-map” to be extensible by just about anyone at anytime.

Time for lunch.

References

Dutton, W. (2010). Networking distributed public expertise: strategies for citizen sourcing advice to government. One of a Series of Occasional Papers in Science and Technology Policy. Retrieved from http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1767870

Nesbitt, K., Sutton, K., Wilson, J., & Hookham, G. (2009). Improving player spatial abilities for 3D challenges. Proceedings of the Sixth …, 1–3. doi:10.1145/1746050.1746056

Putnam, R., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 4–15. Retrieved from http://www.jstor.org/stable/1176586

Riel, M., & Polin, L. (2004). Online learning communities: Common ground and critical differences in designing technical environments. In S. A. Barab, R. Kling, & J. Gray (Eds.), Designing for Virtual Communities in the Service of Learning (pp. 16–50). Cambridge: Cambridge University Press.

Trying out a new writing process

It’s Wednesday, that means writing day. Time to continue work on a paper that’s been mentioned previously. In this process I’m exploring being open about the writing process and attempting to create a collection of artifacts (evidence?) leading up to the final paper. This post documents today’s steps with a particular focus on first steps with a new Smallest Federated Wiki (SFW) enabled writing process that fruitfully merges with use of this blog.

An idea SFW enabled writing process

Perhaps it’s my programming background, but when I start writing a paper or a presentation I start by developing the structure. The sequence of ideas or points that I want to make in the paper. Once I have that structure in place, once I know what I want to say, I find writing that much easier. Of course the structure will change over time, but having the outline of ideas helps.

In the past I’ve used a range of adhoc methods for developing this structure. Ranging from the purely mental approach of reciting and reframing it in my head through to drawing upon artifacts such as pen & paper, text files edited with vi, and Word documents. As the title suggests the plan here is to use SFW. To create a page/article in SFW that consists of a list of ideas/concepts. Each of those ideas/concepts are separate pages in SFW.

The idea being that the writing process will be a sequence of reading, thinking and writing about each of those ideas/concepts. Of fleshing out each of the pages for the single ideas and perhaps moving them around, removing some and adding others. At some stage when I’m happy with the raw material on SFW, the idea will be translate that into Word or some other editing software.

On the plus side, this approach seems to offer better support (the linking of ideas and their separate development, versioning at a paragraph level, enabling others to build on what I’ve written) for a writing approach I’m comfortable with. However, it’s also likely to offer some challenges and misfits. For example, I found it difficult to come up with labels for the ideas/concepts that make up the paper that are both meaningful to me (and the writing process) and general enough to encourage/enable others to contribute and build upon those ideas.

If other people do contribute on build on those ideas, what implications might this have for authorship? At the moment, I don’t see the SFW ideas being in the form of the final paper. They will be more general. When I move to the word processing software I will re-write/write my own version, but it will likely be influenced by the input of any other contributors.

I also wonder about the affordances of the SFW interface for writing. i.e. the level of technical support it offers for entering and organising large amounts of text and my understanding of how best to leverage that. There’s still not the level of “fluency” (I feel somewhat dirty using that word) I’d like.

Associated with that I wonder about the question of citation management. Important to academic writing, but not directly supported by SFW. I’ll have to develop some practices and wonder how those will scale, especially as I move into the writing of the formal paper.

Time will tell.

Case studies in relevant journals

The next phase of writing is to start filling in some of the ideas in the paper structure by looking at the literature. The first area I’m looking at will be the question of case study research in the journals I’m targeting and also around the question in the paper (what’s involved in the reality of trying to develop high quality learning environments in higher education? Does that explain why there’s limited widespread quality?) The point is to identify prior work to build on and learn from.

Raising the questions of how to best search through articles in specific journals. Does Google scholar support this or do I have return to using the old style (horrendous) “library database search”. Ahh, appears the institutional library has moved a step beyond some of my memories from years ago and my local collection of papers is revealing some tidbits.

Time to start sifting and reading.

Progress made. To early to tell how it will go.

At least one of the papers I skimmed resonated with a negative local experience and generated a rant.

Barriers to higher education technology adoption: Digital fluency or usefulness?

Motaghian et al (2013) in talking about “web-based learning systems” (LMS) conclude that (p. 167)

perceived usefulness was the most important influential factor on instructors’ intention and their actual use of the systems (adoption)

This is seen as important since earlier they’ve argued (p. 158)

despite the emerging trend of using web-based learning systems to facilitate teaching and learning activities, the number of users of web-based learning systems is not increasing as fast as expected (Wang &Wang, 2009). Eventually, while e-learning has been promoted to various levels of users, the intention to continue using such system is still very low. Although initial acceptance of e-learning is an important first step toward achieving e-learning success, actual success still needs continued usage (Lee, 2010). However, since the web is a new medium (for educators and learners alike) for course delivery and learning, it is not well known which mediating and moderating factors in the online environment contribute more to its acceptance and use (Sanchez-Franco, Martínez-López, & Martín-Velicia, 2009)

They found that how useful an instructor perceived the institutional web-based learning system to be, was the most important factor influencing use.

They found that “perceived usefulness (0.50) contributed two times more to intension to use than the perceived ease of use (0.25)” (p. 66) and concluded (emphasis added) (p. 66)

As a result, instructor’s perceived ease of use might not be as important as instructor’s perceived usefulness in this context. Thus, instructors will be more likely to continue to use the system if they consider it useful. Hence, instructors’ requirements should be taken into consideration when developing web-based learning systems (Wang & Wang, 2009).

What might that suggest about the idea from the 2014 Horizon Report that the #1 barrier to higher education technology adoption is the low digital fluency of academic staff?

Might it suggest that low levels of system adoption says more about the usefulness of the technology, than the fluency of the instructors?Might it suggest that the requirements of instructors aren’t being taken into consideration in their development?

Given my recent experience (at the same time as reading this paper) with institutional approaches to e-learning, I know how I’d answer those questions.

What about you?

Note: In talking about this I’m generally focusing on how institutions implement these systems (e.g. Moodle), rather than the development of those systems (e.g. Moodle). I think institutions (at least the ones that I’ve experienced) are particularly incompetent and producing systems that I would perceive as useful.

The importance of bricolage

Not to surprisingly, this observation has me thinking about the differences between the SET (in their way) mindset used by most institutional e-learning and the (breaking) BAD mindset used by people like myself.

A SET mindset is takes a Strategic approach to deciding what work gets done. It’s focus is on achieving the institutional plan or vision. For example, making sure that every course uses the standard look and feel template. Instructors’ perceived usefulness of the system is not the prime concern of a SET mindset.

A BAD mindset uses bricolage as a way of deciding what work gets done. Ciborra (1992) defined bricolage as the “capability of integrating unique ideas and practical design solutions at the end-user level” (p. 299). Bricolage is about solving the problems experienced by users. Bricolage focuses on enhancing perceived usefulness.

So, would the SET mindset or the BAD mindset contribute to greater levels of adoption?

Seems like a no-brainer to me.

References

Ciborra, C. (1992). From thinking to tinkering: The grassroots of strategic information systems. The Information Society, 8(4), 297–309.

Motaghian, H., Hassanzadeh, A., & Moghadam, D. K. (2013). Factors affecting university instructors’ adoption of web-based learning systems: Case study of Iran. Computers & Education, 61, 158–167. doi:10.1016/j.compedu.2012.09.016

What might the 3 levels of organisational culture reveal about university e-learning

I have fulfilled my organisational duty and attended and participated in a 3 hour workshop intended to achieve some level of shared vision within the organisation. As always I remain cynical about likely impact such sessions will have on the organisation and my experience of it. There was, however, some benefit in making me aware of Schein’s three levels of organisational culture (apparently from this book) and summarised in the following table (and more on wikipedia).

Schein's Model.JPG
Schein’s Model” by ShiraraeOwn work. Licensed under CC BY-SA 3.0 via Wikimedia Commons.

Three levels of culture
(Adapted from Schein, 1992, p. 24)
Level Description
1. Artifacts Visible and feelable structures and processes

Observed behaviour – difficulty to decipher

2. Espoused beliefs and values Ideals, goals, values, aspirations

Ideologies

Rationalisations – may or may not be congruent with behaviour and other artifacts

3. Basic underlying assumptions Unconscious, taken-for-granted beliefs and values – determine behaviour, perception, thought and feeling

An idea is that the artifacts are one avenue for exploring the espoused beliefs and the underlying assumptions that inform the organisational culture.

Hence my question, what do the artifacts associated with organisational e-learning say about the organisational culture of those institutions? What are the espoused beliefs and underlying assumptions that inform that culture? Are there any contradictions?

What follows is a quick application of this to my next task – starting the preparation of my course site for the next semester. This is a simple exercise not an in-depth analysis and certainly informed by the barest of familiarities with a very specific view of culture. But on the face of it, this exercise strikes me as as useful lens. (It would appear that @leahpmac already done some work around “culture and e-learning”. It’s a small world.)

1. Artifact – Standard course design

The artifact I need to deal with is the new look and feel for the institutional LMS. This means that every course will not only look the same, there are some expectations about what is expected to be on the course site. For my course, this means it will look something like the following

Home page

Everything at the top of the page and in the left hand column is part of the new, standardised look and feel. Everything under the “Welcome to EDC3100: ICT and Pedagogy” heading is what was copied over from the last offering of the course. Hence the “Right now:” message suggestion the course has ended. The following is what the 2014 course site looked like.

edc3100 2014

Broadly speaking the changes involved in this project were

  • Changes in the branding/look and feel.
  • More moved into the reduced banner.
    Explicit mention of the time at USQ, notifications, my courses, useful links, course administration etc moved into the banner which takes up just a bit less vertical space. These look like good moves, but the breadcrumbs continue to include the unnecessary listing of the faculty to which the course belongs. Don’t think the students care about the faculty and it takes up a fair bit of space.
  • The removal of any ability to have a right hand column.
    Only two columns, not three now.
  • AN “expand all/collapse all” option for the main content area.
    I assume this is an attempt to address the scroll of death problem with Moodle. Wonder how much of this is implemented with standard Moodle and how much local customisation?
  • The addition of highly visible menu as the first element in the left hand column.
    These appear to be divided up into two broad categories of items

    1. Links to existing Moodle features (Forums, Resources, Calendar, Participants); or,
      As the labels suggest, these all point to equivalent Moodle functionality. e.g. Forums links to the Moodle service that lists all forums etc.
    2. institutionally specific pages that are semi-integrated with institutional data sources (Study Schedule, Teaching Team, Assessment).
      The basic model for these pages is to know about institutionally specific information such as the dates associated with weeks of semester, due dates for assignments, and the staff teaching into the course. Such information is drawn from institutional databases and combined with some capability to use the Moodle HTML editor to add additional text. For example, the study schedule will fill the date for a Week 1 and teaching staff can manually edit information such as the name of the module for that week.

2. Espoused beliefs and values

As expressed in the support resources, the rationale for this new look and feel include

  • similar experience;
    The desire that students have a similar experience regardless of the course.
  • findability.
    That students are able to find the information they need easily.

The support video for the assessment tab also proposes that the assessment tab “will be very useful for your students”.

3. Assumptions

Obviously I do not know what assumptions these beliefs are based upon, but the following perhaps are not a million miles away

  • Consistency is generally a good thing for learning.
    Given the institutional strategic plan putting some significant weight to personalisation, creativity and innovation, having everything the same doesn’t seem appropriate. Insights from research around learning, teaching, and educational technology would seem to support that. e.g. some of the points from Chris Dede (Harvard Professor of Education) mentioned in this post.
  • A consistent look and feel will make information more findable.
    Wikipedia suggests that findability involves a bit more than just user interface design. Especially when the user interface design is only really helping users locate very specific bits of information. i.e. The new look and feel does make it easy for students to find the forums, assessment, study schedule, and teaching team for a course. Good. But what about the content included in the resources? More on this below in “What about the resources”.In short, if findability is a concern, install a search engine!
  • That it’s possible to have a consistent look and feel across the diversity of courses in an institution.
    The new look and feel does have some features that allow for flexibility. Even though this does raise questions about the consistency espoused belief. If no information is entered for assessment or study schedule the students won’t see those options in the menu. In addition, the study schedule page provides some flexibility in terms of how many columns form the study schedule and the column titles. Allowing individual courses to substitute in the language they use.
    However, there’s a limit to how far this goes. More on this below in “Grouping the weeks”.

An assumption that appears to underpin this new look and feel is that the focus is student centered. The aim is to enhance the student experience. Now that’s a good aim, perhaps the best aim. But the follow on assumption in this case is that teaching staff aren’t capable of using the online environment to enhance the student experience and that the institution needs to do something.

From 1997 through 2004 I helped design, implement and support one approach to an institution doing something about this.
Since then I’ve made the argument for this. However, there are two important points missing from the “new look and feel” at my current institution, they are

  1. the ability to opt out; and
    There will always be academics who can and wish to create their own course sites. The approach provided the opportunity for academics to do this.
  2. an adopter-focused and emergent development approach to the new look and feel.
    i.e. it’s not sufficient for a project team to design the new look and feel and roll it out. There will be inevitably problems with the look and feel and there will be some really good ideas about how to enhance it that emerge from on-going use. How students and staff are using the new look and feel needs to be closely watched and those insights used to continually develop the new look and feel to solve problems and enhance it.

The absence of these points from the new look and feel suggest that there is an underlying assumption that there is nothing to be learned from the teaching staff and their experience. It’s a prime example of the “do it to” and perhaps “do it for” paths and an apparent avoidance of the “do it with” path (Beer et al, 2014).

Interestingly, I’ve just found the following slide on Flickr that purports to represent Schein’s cognitive transformation model for analysing organisational cultures. I’m guessing this was the basis for the consultant/facilitator. What I find particularly relevant to the specific decision is the circle around the outside labelled “organisation iteratively adapts” which I see as resonating with the adopter-focused and emergent development approach mentioned above.

3 Levels of Organization Culture (Schein by MizzD, on Flickr
Creative Commons Creative Commons Attribution-Noncommercial-Share Alike 2.0 Generic License   by  MizzD 

What are the limitations of the new look and feel?

The following explores the dissonances that exist between the new look and feel and the approach I use in my course.

What about the resources?

The following image is a partial screen shot of the “Resources” page for my course.

Resources

This partially illustrates that my course is designed so that each week contains a learning path. A collection of activities and resources that all students are expected to work through. The “resources” page only shows the resources, but at least it does show you that there are quite a few Moodle books amongst those resources.

A Moodle book is essentially a collection of web pages. The Moodle books are used to structure student learning around a task or some related concept. For example, the first book in Week 1 is titled “Setting up” and is designed to help the students set up Diigo, their blog and Twitter. The next book explains the first required learning task for the course – introducing themselves.

Each book is typically at least 3 or 4 web pages long. A quick visual count reveals almost 50 separate Moodle books in the learning paths. Some can cover some important concepts. Concepts that the students will wish to revisit later in the semester. In particular, some books give advice about assessment. It’s not unusual for students to ask “Where did we talk about *insert topic*“.

Nothing in the new look and feel will help students find this information.

A search engine would.

It would be nice not to have to implement this kludge again to enable the search.

Duplication and confusion

It would be interesting to find out the thinking behind promoting the “Resources” link into the menu for the new look and feel. I assume the aim (in line with the espoused beliefs above) is to make it easier for students to find the resources and that this is important for learning.

However, I wonder if it’s going to create some duplication/confusion, especially given the design of my course.

The image above shows part of the resources view for my course site, including most of the initial resources for Week 1. The following image shows the learning path for week 1.

Week1

The resources page offers essentially the same view as the learning path, but it misses two components. First, it doesn’t include the activities (e.g. the discussion forums “Share you introduction” and “Where you fill in the blanks” are missing). Second, the headings are missing. These are used to group the resources and activities into meaningful groups.

The presence of the resources link doesn’t appear to add any value to students and appears likely to create some confusion.

Grouping the weeks

Adding a study schedule page is potentially a useful addition. Something that isn’t present on may sites. Some students may find this useful. That’s the reason why I’ve had a study schedule in my course site since I started.

Take a look at the “Course content” box in the middle of the page below. Do you see the link “study schedule”?

Home page

The following image shows part of the study schedule I’ve created. A problem I have is that there are some features of this study schedule that the new look and feel won’t support

  1. grouping weeks by modules;
    In the following, Module 2 includes weeks 4, 5 and 6. The new look and feel doesn’t support this grouping. This is problematic because the course is designed to have four modules and it’s a good thing that the study schedule clearly shows these modules.
  2. assignments as separate rows; and
    The submission date for assignment 2 is quite clearly shown in the study schedule by an entire row of a different colour. The new look and feel’s schedule embeds this as part of the a cell in the week’s row.

Not major problems, but illustrations of how a consistent approach to course design breaks down when it meets the design decisions made by individual teachers. If those design decisions are bad, there may not be a problem. But what if those design decisions are valid? Is it appropriate that those design decisions should be thrown our and the course revert to the norm?

studySchedule

Limited assessment information

Just under “Study Schedule” in the “Course Content” box above you will see a link labelled “Assessment”. i.e. my course site already provide a range of information about course assessment. This is again a case of the new look and feel duplicating what I already do, and doing so in a way that loses functionality.

The new look and feel assessment page does have some advantages. For example, it allows you to create cohort specific assessment information that is only seen by that cohort. The trouble is that I don’t do that in my course. So no value for me.

The new look and feel’s approach to assessment creates a single page for assessment. Everything about assessment for a course on a single page. This is a problem as the following shows.

Assessment

Can you see the “Table of Contents” heading in the left-hand menu? That’s the start of the list of information I provide on Assessment. It includes the following

  • Assessment (1 and a bit pages long)
    Overview of course assessment, due dates, percentage etc. What you see in the above image.
  • How to request an extension (almost 2 pages long)
    Some FAQs about extensions and details of how to ask.
  • Learning Journal (almost 3 pages long)
    The learning journal is a core part of the learning design (and assessment) of the course. It’s new to students, this section offers an explanation of the learning journal.
  • Problems with the learning journal (about 2 pages long)
    FAQs about the learning journal, particular common problems.
  • Assignment 1 (3 and a bit pages)
    Detailed description of assignment including submission process and the rubrics.
  • Assignment 1 Questions and Answers Video (less than page)
    A video/screen cast answering FAQs about assignment 1.
  • How to submit assignment 1 (less than a page)
    Another video showing how to submit the assignment.
  • Assignment 2 (about 3 pages)
    Detailed description of assignment including submission process and the rubrics.
  • Assignment 3 (about 4 pages)
    Detailed description of assignment including submission process and the rubrics.
  • But I’m not going on Professional Experience? (a page and a bit)
    Assignment 3 is linked to professional experience. In some circumstances students aren’t going on Professional Experience. Describes what those students do.
  • How to query the marking (a page)
    Describes the process students should use to query the marking of their assignments.

That’s a total of about 22 pages (as you might see the new look and feel uses a larger font and white space) of information that under the new look and feel would appear to have to go onto a single page. Horrendous for me to create and worse for students to actually find and use any information.

With the above I use the Moodle book plugin to create and manage this collection of information. The Moodle book plugin also provides a nice way for students to print this information. Either the entire book or selected “chapters”. This “nice way” includes removing all of the additional web interface elements.

The new look and feel does attempt to implement something like this “nice way”, however, it’s a generic web approach that leads to overlapping of real content with basic web navigation (at least on my Mac).

Problems with Moodle books

Exploring the difference with assessment has revealed some additional problems around the use of Moodle books and the new look and feel. The following image is that Assessment overview page in the old course site as it would be seen by students.

Old Assessment

In the above image, look for the following components

  1. Table of contents; and
    This is in the top left hand corner. It’s the ToC for the Assessment book and shows all of the components. It aids findability. Students can see all of the chapters in the book.
  2. Book administration.
    This is in the bottom right hand corner. It has two important links: “Print book” and “Print this chapter”. These are the links that allow students to print versions of the book/chapters with just the content (the middle column).

Now, look below. This is the same assessment book in the new look and feel. What do you notice about the “Table of Contents” and the “Book administration” components?

Assessment

The two problems that I see are

  1. Table of contents is partially below the fold; and
    Due to the new look and feel’s use of a larger font, more whitespace, a single column on the left-hand side, and retaining the standard menu at the top of the left-hand menu the Table of Contents for the book gets pushed down. So that large parts of it aren’t visible.
    I should note that I have a large monitor and keep my browser windows open in a longer format than most. There will be some students for whom the Table of Contents will not be visible.
  2. There is no “book adminsitration” component.
    Actually, there is. It just doesn’t appear in the image above and I’ve only just know found where it is located after a concerted effort to find it. i.e. I knew it had to be there, so I went looking. The following image shows the “Forums” page in the new look and feel. IN the black bar at the top of the page you should be able to see “Forum administration”. When viewing a Moodle book this is where “Book administration” will appear.

DiscussionForum

What do I need to do?

Based on the above, here’s a list of questions to answer

  1. Can I change the study schedule and assessment links to my existing approaches?
    Happy to stay with the menu and the new look and feel, but would prefer that the menu link to the relevant Moodle books.
  2. Can I remove the resources link?
    I don’t think it adds any value in my course and is potentially going to cause confusion. So can I remove it.
    Question: I wonder whether the organisation has done any analysis of student usage of the course sites over Semester 3 (when they’ve been using the new look and feel)?
  3. Can I implement a macro/API system to insert USQ information?
    The one really useful addition to the new look and feel is the integration with USQ systems to pull information (e.g. dates for each week, assignment due dates etc). The problem is that this functionality is only available within the assessment and study schedule features from the new look and feel. As above, I don’t want to use these.
    However, it would be really useful if there were a combination API and course site wide macro facility that would allow me to enter something like the following in the HTML for my course

    Assignment 1 due: <div class="assignment_1_due_date" course="edc3100" offering="2015_1"></div>

    and have it automagically replaced with the appropriate due date.

The API is a step too far, but some kludges with javascript might make the macro possible.

  • How best to explain to students how to best use Moodle books in the new look and feel?
  • What can and might I do around providing a search engine for the course site?
    I feel this may be a step too far for this year.

 

If I am to be a good organisational citizen, then in answering the above questions I should be raising these questions through the formal support mechanisms and waiting for them to identify whether these are possible and allowed. (I fear that the latter is more likely to be the real problem).

Of course, there are also the possibility that some of the above can be implemented through a bit of bricolage.

Lessons from Schein’s 3 level of organisational culture?

Arguably I’ve established above that organisational culture contains strong assumptions about consistency equating to findability. That it is possible to employ a consistent approach across all courses in an entire university. I think I’ve established from the above that there are problems with these assumptions. I also think that I’ve illustrated that the new look and feel is (or at least appears to be) suffering from the absence of on-going iterative development. i.e. it’s not learning from itself.

In the workshop a slide was handed around that showed artifacts, espoused beliefs, and assumptions as three vertically arranged boxes (typical Powerpoint). Between assumptions and espoused beliefs, and espoused beliefs and artifacts, there were two arrows, each pointing from one box to the other. The suggestion being that assumptions influence espoused beliefs and espoused beliefs influence assumptions. Similarly for artifacts.

As we’ve argued in this paper, I believe that the organisation works on the assumption that its digital artifacts – such as the new look and feel – are established. i.e. they can’t be changed by anyone very easily and certainly not be anyone who hasn’t been approved via the appropriate governance structure. Hence the arrow suggesting that anything should be changing the artifact is somewhat attenuated when it comes to digital artifacts in enterprises.

However, as we’ve argued in another paper digital technologies are protean. They are flexible and changeable. Some more so than others. For example, phone apps are hard to change unless you’re the developer. But the web environment is definitely protean. Suggesting that the ability to change the artifact is possible from cultures that don’t hold the assumptions or espoused beliefs of the dominant organisational culture.

Of course, the problem is making changes to artifacts set up with the assumption that they are established. Can be harder than you think.

Learning about case study methodology to research higher education

The following is a summary and some thoughts on Harland (2014). The abstract for the paper is

Learning about teaching through inquiry is a sound practice for professional development and the university teacher-as-researcher is now commonplace. With the proliferation of inquiry-based postgraduate programmes in university teaching, more academics from across the disciplines are learning new ways of doing research. In this paper, I draw on 10 years’ experience of teaching research methods in higher education. I teach on a one-year part-time course that aims to help academics change their practice and contribute to the theories of teaching in higher education through publication of their work. The preferred research method is case study and there are four questions that both inexperienced and experienced participants can find challenging: what is the potential of case study; what forms of data are acceptable; when does analysis stop and what makes a quality case study? I conclude with a set of recommendations for the new researcher aimed at enhancing the quality of research. Suggestions include properly integrating existing theory into published work, avoiding positivist analogues to judge research, using multiple methods in research design and avoiding writing descriptively or without a critical audience in mind.

My interest

As outlined previously, I have to write more journal articles. The current paper idea I have goes under the following working title “BAM, BIM, Blogs and Breaking BAD: What does it take to create quality e-learning?”. It’s currently conceptualised as a case study of the development and use of BAM and BIM to support the use of individual student blogs from 2006 through 2015. The basic argument is that it’s no surprise that most e-learning is not that great, given the difficulty of doing anything decent within the current institutional mindset around e-learning. The idea is to draw on the Breaking BAD paper, a presentation to MoodleMoot and various other publications round BIM/BAM over the years.

Given it’s a case study and they have limitations, it’s probably a good idea to be able to write up the method in a way that ticks all the right boxes. Hence my interest in Harland (2014).

Thoughts

Case study is a dominant/well accepted method. Tick.

How often does Computers and Education accept case study work?

Points out some interesting points for consideration by early case study researchers in higher education. Not a “how to” guide such as Baxter and Jack (2008)

Summary

Introduction

Researching own teaching practice one way academics learn to teach. Teacher-as-researcher, link to high schools and Boyer’s SoTL. Defines “dual researchers” people rsearching both their discipline and the teaching of that discipline.

Author from hard sciences. Research in academic development didn’t follow the science rules.

cites Tight (2012) that qualitative inquiry remains dominant research method in higher ed journals.

As a reviewer, author finds most articles are sub-par. Case study used in most articles read.

# of case studies published in four higher education journals: 2007-2012
adapted from Harland (2014, p. 1114)
Journal Location Case study Conceptual study Total #
Higher Education Europe 344 181 525
Studies in Higher Education Europe 238 62 300
Teaching in Higher Education Europe 176 111 287
Higher Education Research & Development Australasia 146 101 247
Total: 904 455 1359

“Case study consists of empirical inquiries of single cases that are contextually unique (Stake, 1995)” (Harland, 2014, p. 1114) – my emphasis added – typically addressing something of interest to the authors professional practice. Has instrinic value to those that benefit from the professional practice, but can also contribute to “the theories and practices of higher education”

There are four sticking points in learning case study research methods

  1. What is the potential of the case study?
  2. What forms of data are acceptable?
  3. When does analysis stop?
  4. What makes a quality case study?

These are used to structure the rest of the paper.

Has a para which appears essentially the research method para. Autoethnography is used. The four sticking points are addressed through a personal narrative.

What is the potential of the case study?

Specificity of case study research seen to limit contribution to theory. But that type of certainty of knowledge is very techno-rational. Case study inquiry involves individuals/teams interpreting data. Requiring new standards of judgement (Flyvbjerg, 2006) who contrasts rule-based and case-based knowledge. Case-based is always context-dependent.

As I see it, no two practice contexts are ever genu- inely the same and so rules and deterministic models for guiding thinking and action are not that useful. (p. 1115-1116)

Case study research cannot be truly replicated given the uniqueness of context, but it can be learned from. What each reader may learn will differ.

While its possible to generalise from case studies (Denzin, 2009), it’s unusual. Though it is argued that cases provide an opportunity for generalisation.

Case study research seen as better for generating hypotheses than theory building (Flyvbjerg, 2006) – depending on the definition of theory. Options include

  1. Explanatory and predictive of cause & effect and thus can direct action
  2. pragmatist perspective that has theory/practice intertwined. People generate theories to seek meaning in practice. A form of personal theory building.

Theoretical relevance enhanced if existing theories are integrated. Through which contribution can be made through new interpretation of data. “Existing theory should be seen as an integral part of the case” (p. 1116)

What forms of data are acceptable?

“case study may rely on multiple sources of evidence and be practiced as multi-method research (Denzin & Lincoln, 1994)” (p. 1117) as long as it helps answer the research question. i.e. it can use quantitive research methods. Apparently a surprise to some.

Tight (2012) found only 5% of 440 published articles in 15 higher education journals used a mixed quantitative/qualitative methodology.

What does analysis stop?

Outcomes of any analytic technique will depend on intentions, background knowledge, cognitive processes, mindset etc. Hence analysis is recursive.

Minimise the time between collecting data and writing the research account. “disciplined writing seems to be the most essential part of the analytical process” (p. 1118).

No genuine endpoint.

What makes a quality case study?

Better to engage wider theories than just describe practice. But not sufficient

Quality case research:

  • requires imagination (Dewey, 1938)
  • requires creativity (Morse, 1995)
  • must bring the reader as close as possible to the experience (Fossey et al, 2002)
  • provide conceptual insight (Siggelkow, 2007)
  • should be believable, which requires coherence and provide new theory and instrumental utility (Eisner, 1991)
  • “potential to create an impact on the field of practice” (p. 1118)
  • have something important to say
  • well structured and clearly writte

Argues that a case study should enable someone to learn from it.

Author explores the impact (from this measure) of one of his case study publications. Somewhat sobering results.

Conclusion

Summarises the key points to be “attentive to” and “cautious of” against the four challenges and some more general comments e.g. research must fulfil its purpose and this needs to be known before time to help align process and outcomes.

Does make the point that case study is a form of learning and that this can be seen in daily practice, more so than in research articles.

Also points out that publications on case study methods are “often complex or underpinned by unstated assumptions, following a procedure is never straightforward” (p. 1121). Case study research does allow you to learn from experience, so these methods should be seen as guidelines.

References

Baxter, P., & Jack, S. (2008). Qualitative case study methodology: Study design and implementation for novice researchers. The Qualitative Report, 13(4), 544–559.

Harland, T. (2014). Learning about case study methodology to research higher education. Higher Education Research & Development, 33(6), 1113–1122. doi:10.1080/07294360.2014.911253