Category Archives: addie

The alignment project as leadership

The following signals a slight change in direction around the curriculum mapping project. First, the project is now going under the label “alignment project” (curriculum mapping is just one aspect of the project). Second, the project is likely to be re-framed as an application for an ALTC leadership grant. This post is an attempt to begin this re-framing. It’s really just thinking out loud.

As a result, I am very interested in suggestions and criticisms. In terms of suggestions, I’m particularly keen for insights onto better/alternative theoretical frames. It has become a bit confused as I’ve tried out different lines of approach.

Note: In the following a “course” typically means the smallest unit of teaching offered by an institution. A “program” is a group of courses that form a degree or perhaps a major.

Summary of the alignment project

Alignment is an increasingly core component of teaching in Australian universities. At a basic level, alignment is where the learning resources, learning activities and assessment of learning all align with the stated outcomes or aims of teaching (in a course/unit or program/course). Such alignment is a core component of various “movements” within university learning and teaching including: graduate attributes, quality assurance, and improvements in learning and teaching. However, consideration of alignment is not a regular, everyday part of teaching or learning at universities.

The majority of academic teaching practice involves the teaching of an existing course, one the academic has usually taught before, and as such most “teaching practice” revolves around making minor modifications to material or content (Stark, 2000). Academics are not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988). Alignment is most considered during the development of new courses, major overhauls of existing courses or in response to external quality assurance needs.

The following two sub-sections try to summarise the alignment project. The first is a more concrete description, the second more abstract or theoretical. The thinking behind this project has changed many times, the following are likely to change. Suggest away.

Process and intent

As currently thought, the alignment project can be described as four main tasks:

  1. Modify Moodle to allow mapping of alignment.
    Moodle has been chosen because it is the institutional e-learning system at both institutions that are part of the project. As such, support for Moodle is embedded into the institution and will continue to be supported. In addition, Moodle is an increasing part of the everyday practice of teaching academics. Lastly, Moodle, for a number of reasons, is very flexible and easy to modify and any modifications made could be usable by other institutions. The point of this project is not the Moodle modifications, it’s simply the best solution for embedding these changes into the institutions.

    The changes will focus on enabling the alignment of outcomes (be the course learning outcomes, program learning outcomes, those from accrediting bodies, graduate attributes etc) with the assessment, activities and resources within a course site. Having this functionality is a foundation for the rest of the project.

  2. Work with teaching staff to map their courses.
    Mapping the alignment of a course within Moodle is not going to be straight forward. Teaching staff are likely to be busy and may not be entirely familiar with the concepts around alignment and mapping. The content and design of a Moodle course site may not be currently appropriate in terms of developing a useful mapping of alignment. Using the alignment mapping functionality added to Moodle may not be straight forward.

    For these and other reasons staff from the L&T support services will have to work collaboratively with academic staff to overcome these problems. This will be the first cycle of action research and will identify specific problems and insights into potential solutions.

  3. Work with teaching staff to embed alignment into everyday practice.
    Once the initial mapping of alignment is complete, the focus moves onto helping academics maintain and reflect on the level of alignment within their course and programs. On helping them embed alignment considerations into everyday practice.

    This is the second cycle of action research and will likely include the development of models, tools and processes that address questions such as:

    • How do you encourage reflection and action based on the everyday consideration of alignment?
      Identify the support, processes and tools do teaching academics and program leaders need to encourage and enable reflection and action?
    • How do you measure and give feedback upon action based on the everyday consideration of alignment?
  4. Develop on-going and embedded institutional processes that take the lessons learned from the everyday consideration of alignment and use it to remove barriers within the institutional context.

Theoretical perspective

The alignment project is seeking to take the first steps towards what Biggs (2001) called the reflective institution. It seeks to do this by modifying the institutional systems around learning and teaching in ways suggested by Biggs (2001). These are:

  1. Make obvious the quality model.
    Most institutions espouse the theory of alignment, however the teaching systems and processes employed by institutions do not make this theory explicit. The first step is to modify these systems so that a focus on alignment is made explicit as a part of every day teaching practice.

    This is partly achieved through the modifications to Moodle to enable mapping of alignment. But more importantly it is achieved through the changes in L&T support, systems and processes that support academic in using those new Moodle capabilities. In part, these changes are the next step.

  2. Provide appropriate support for quality enhancement.
    Further modify these systems and process to enable and encourage teaching staff to reflect and improve their teaching through a focus on alignment. The modifications build on the changes in the previous stage to enable this support to be highly contextualised to everyday teaching practice. A focus on helping in what academics do most often, the fine-tuning of existing courses.
  3. Inform the quality feasibility process.
    Quality feasibility is the removal of factors in the teaching environment that are not conducive to good L&T. The everyday consideration of alignment will identify a range of barriers in the institutional setting, many of which will require the engagement of institutional leadership to remove.

ALTC leadership grants

The guidelines for the ALTC leadership grants scheme describe the grants as being for (my emphasis added)

projects that build leadership capacity in ways consistent with the promotion and enhancement of learning and teaching in contemporary higher education, and which reflect the ALTC’s values of excellence, inclusiveness, diversity and collaboration, and its commitment to long-term, systemic change.

The scheme has three priorities which can be summarised as being focused on: institutional leadership capacity building; disciplinary and cross-disciplinary leadership; and building on earlier projects. The alignment project seems to be best suited to Priority one

institutional leadership to enhance learning and teaching through leadership capacity-building at the institutional level.
  • Funding range: $150,000 to $220,000
  • Project duration: up to 2 years

In completing a lit review around leadership for the ALTC, Southwell and Morgan (2009) make the observation that

Leadership for Excellence in Learning and Teaching Program is expected to be a ‘demonstrable enhancement of learning and teaching through leadership capacity building’

What is leadership?

It seems required when discussing leadership to make the observation that “leadership eludes comprehensive definition” (Southwell and Morgan, 2009). Southwell and Morgan (2009) reference Marshall (2006) and Jameson (2006) as folk who have made that observation. Having made this claim, the idea is that you then define your familiarity with the broad array of perspectives, understandings and definitions of what leadership is. I’ll postpone that bit for now.

Southwell and Morgan (2009) draw on Leithwood and Levin (2005) who suggest that the core of most conceptions of leadership are two functions generally considered to be indispensable:

  1. Direction-setting: helping members of the organization establish a widely agreed on direction or set of purposes considered valuable for the organization; and
  2. Influence: encouraging organizational members to act in ways that seem helpful in moving toward the agreed on directions or purposes

Leithwood and Levin (2005) arrive at these two functions by adopting a definition where “the primary effect of organisational leadership would be significant change in a direction valued by the organisation”. In defending their definition or understanding of leadership, Leithwood and Levin (2005) agree that this may not be a precise definition, but that attempts to too narrowly define a complex topic like leadership is “more likely to trivialise than help bring greater clarity to its meaning”.

How the alignment project fits

Taking the emphasised phrases from the above purpose of the ALTC projects, I’d suggest/argue that the alignment project fits in the following ways:

  • leadership capacity;
    In terms of the above set of functions, the project is aiming to build into the systems and processes of each host institution the capacity to make consideration of alignment an every day part of practice. It helps establish a widely agreed direction for L&T and helps influence organisational members in moving towards that agreed direction.
  • promotion and enhancement of L&T;
    The educational literature is replete with evidence that consideration of alignment changes the conception of L&T held by academics and that it also results in improvements in student learning outcomes.
  • contemporary higher education;
    The project recognises and seeks to fulfill the increased requirements for accountability from a range of diverse source, however, it seeks to achieve it in a way that offers significant greater benefits that existing methods. As part of this the application seeks to engage with the on-going argument over centralised or devolved L&T support services by aiming for a focus on an approach to L&T support services that seeks to contextualise such support into the every day practice of teaching academics.
  • inclusiveness, diversity and collaboration; and
    The action research process suggested for use by the project is largely based on recognition within the project that engaging with the full diversity of higher education is essential. It recognises that this diversity will result in different approaches and benefits and that the process needs to enable this to happen. Similarly, collaboration is seen as essential to the project. Not just in the process used in this project, but in the aims of the project. An important aim of the project is to increase the collaboration around consideration of alignment in teaching.
  • long-term, systemic change.
    The project aims to embed consideration of alignment into the everyday practice of teaching staff. i.e. the aim of the project is long-term, systemic change. The process and approach being used is designed to achieve that aim.

The alignment project as leadership

Contemporary higher education is placing increasing importance on the concept of alignment in learning and teaching. In terms of quality assessment, program accreditation, graduate attributes and generally improving L&T alignment is broadly seen as a necessary component. However, many university courses are not all that well aligned and one explanation for this is that consideration and discussion of alignment is not a regular part of everyday teaching practice. Alignment is often only considered at the time of course and program reviews or accreditation.

The aim of the alignment project is to build leadership capacity into the system and processes of education within a university so as to encourage and enable effective and informed consideration of alignment as part of everyday teaching. This embedding of alignment into everyday practice then serves as the foundation for a range of other possibilities.

The alignment project is an example of leadership as it is attempting to encourage significant change – in the form of increased consideration of alignment at all levels – that is valued by the individual universities and the broader higher education sector. In addition, there is broad agreement in the education literature that alignment has significant positive effects on student learning outcomes.

The alignment project intends to fulfill the two indispensable functions of leadership identified by Leithwood and Levin (2005):

  • direction-setting; and
    By making considerations of alignment a visible and hopefully key aspect of everyday teaching, there should be an increased emphasis placed on learning outcomes, graduate attributes and other “outputs of learning”. This should encourage and assist academics teaching the same course or in the same program to increase discussion of these outcomes. To increase discussion of the purpose or direction of a course or program.
  • influence.
    A specific aim of the alignment project is to modify the teaching environment so that considerations, discussions and reflection upon alignment are directly encouraged and enabled. It encourages and enables them to think about how to move towards the stated directions or purpose.

Still not happy with that division. First attempt to make it concrete. Another query that arises from this is whether or not this project builds leadership capacity at two or more levels. For example, one approach is that it builds capacity at both the:

  • Institutional level; and
    At an institutional level it enables curriculum/program alignment, but also accountability etc.
  • Instructional level.
    This is the level between student and course/teacher. It makes alignment an more regular component of instruction.

Outcomes

If successful, the project should result in the consideration of alignment should becoming an every day component of teaching practice. This should/could generate the following outcomes:

  • Changes in the conceptions of L&T held by teaching staff;
    Being required to consider alignment encourages a different way of looking at teaching. This should encourage changes in the conceptions of L&T held by teaching staff towards those considered more appropriate.
  • Improvements in student learning outcomes;
    Effective considerations of alignment should increase the alignment within courses. The educational literature suggests that increases in alignment will result in improve student learning outcomes. In addition, it is likely that one of the likely additional outcomes will be making alignment more visible to students. Which should also encourage improved student learning outcomes.
  • Improvements in the quality and timeliness of quality assurance; and
    In terms of demonstrating alignment against outcomes or attributes, current quality assurance practices rely on special “mappings” that are held every few years. By embedding alignment considerations and mapping into every day teaching practice, the there is no longer any need for special “mappings”. “Mapping” information can be generated at any time as it is maintained as part of normal practice.
  • A variety of additional outcomes.
    Embedding alignment considerations as an everyday practice is the foundation of the project. The availability of “mapping” information and the on-going consideration of alignment will generate a range of additional outcomes. However, the diversity inherent in universities and higher education, combined with the fundamental change in everyday practice which this project aims to achieve means that it is unlikely we can predict successfully all of these outcomes.

Theoretical foundations

The ALTC place significant emphasis on applications having a sound and obvious theoretical base. The theoretical work that have informed my thinking about this project, and which may influence the project, include the following.

Project intent or outcome

Biggs (2001) presents an argument for a reflective institution that focuses on prospective, rather than retrospective quality. Having just re-read the paper, it seems to provide a good fit for a theory/model for the overall intent of the project.

The model is based on the idea of three aspects of AQ:

  • Quality Model.
    i.e. an espoused theory of teaching, for Biggs this is constructive alignment. For the alignment project this might be alignment a little more broadly.
  • Quality enhancement.
    A “teaching delivery system” that is designed in accordance to the quality model, i.e. one which encourages and enables alignments. In addition, the teaching delivery system should also have built-in mechanisms to continually review and improve current practice.
  • Quality feasibility.
    A process by which impediments to quality teaching are removed from the “teaching delivery system”.

The “quality model” is underpinned by a large, overlapping and diverse collection of literature from various areas including: outcomes-based quality/evaluation, instructional alignment (Cohen, 1987); curriculum alignment; graduate attributes; and, of course constructive alignment (Biggs and Tang, 2007; Biggs, 1999 ).

Understandings of universities and organisations

My personal conceptions of most organisations, but especially those like universities, are informed by Dave Snowden’s Cynefin framework and complex adaptive system.

Understandings of leadership

This is obviously an area which needs more consideration, beyond the summary given above.

Based on my limited reading, I like the description of “new leadership” (related to distributed leadership) attributed to Fullan (2008), which include:

  • respect of employees, rather than simplistic judgmentalism;
    A specific focus of this project is to help teaching staff consider the alignment of their courses. It is not to judge them. Biggs (2001) makes the point that under the type of reflective institution he outlines, the focus is on the teaching, not the teacher.
  • connecting peers with purpose and ownership;
    By embedding indications of alignment into the LMS the aim is to create connections between other teaching staff, academic leaders and also teaching support staff. The clear purpose is around considering alignment.
  • building employees’ and systems capacity; and
    This is the specific aim of the project. Building into the institution the capacity in both the employees and its systems to engage regularly in consideration of alignment.
  • transparency of practice and results.
    At the very least, the aim of this project is to enable teaching staff within the same program (i.e. group of courses/degree etc.) to see each others practice. To show what is aligned, where (or where not). It opens up the teaching practice to colleagues, hopefully in a way that is not judgmental.

Process

Based on some of the above and below, I’m leaning very much towards an action research process. The practice of L&T within a university is a complex-adaptive system. As we introduce change into the system, the system will change around us and unexpected event will happen. The type of change suggested involves a fairly widespread change in the practice of teaching academics, but also the institution. In addition, as far as I’m aware, no-one else has tried and reported on this type of change, hence it is novel. While driving towards a particular goal, we have to aim on learning as much as we can during the process.

In addition, Biggs (2001) offers the following

action research, a methodology designed precisely to generate and evaluate in-context innovations (Elliott 1991). As a result of engaging in action research, teachers change their conceptions of teaching, and teach more effectively (Kember 2000).

This type of approach also fits very closely with what is known about staff development.
i.e. the current recommendations are that staff development should be as contextualised as possible. My conception of how teaching staff would be helped to consider the alignment of their courses contains a very heavy assumption on contextualised staff development. In fact, the presence of alignment information and the transparency between courses is aimed at helping this support and development to be increasingly contextualised to the every day teaching practice of the teaching staff.

Teaching practice

One of the fundamental models/theories underpinning this project for me is the observations embodied in the following.

How academics design their teaching is not described by a rational planning model (Lattuca and Stark 2009). In part, this is because the dominant setting for academics is teaching an existing course, generally one the academic has taught previously. In such a setting, academics spend most of their time fine tuning a course or making minor modifications to material or content (Stark 2000). Academics are usually not often required to engage in the development of new courses or major overhauls of existing courses (Stark and Lowther 1988).

The practice of most academics does not separate planning from implementation, and rather than starting with explicit course objectives, starts with content (Lattuca and Stark 2009). In the absence of formally documented teaching goals, the actual teaching and learning that occurs is more in line with the teacher’s implicit internalised knowledge, than that described in published course descriptions (Levander and Mikkola 2009). Formal descriptions of the curriculum do not necessarily provide much understanding about how teachers put their curriculum ideas into action (Argyris and Schon 1974)

References

Argyris, C., & Schon, D. (1974). Theory in practice: Increasing professional effectiveness. Oxford, England: Jossey-Bass.

Biggs, J. (1999). Teaching for quality learning at university. Buckingham: Open University Press.

Biggs, J. (2001). The Reflective Institution: Assuring and Enhancing the Quality of Teaching and Learning. Higher Education, 41(3), 221-238.

Cohen, S. A. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16-20.

Fullan, M. (2008). The Six secrets of change. San Francisco: Jossey-Bass.

Jameson, J. (2006). Leadership in post-compulsory education: Inspiring leaders of the future. London: David Fulton Publishers.

Kember, D. (2000). Action Learning and Action Research: Improving the Quality of Teaching and Learning. London: Kogan Page.

Lattuca, L., & Stark, J. (2009). Shaping the college curriculum: Academic plans in context. San Francisco: John Wiley & Sons.

Levander, L., & Mikkola, M. (2009). Core curriculum analysis: A tool for educational design. The Journal of Agricultural Education and Extension, 15(3), 275-286.

Marshall, S. (2006). Issues in the development of leadership for learning and teaching in higher education (Occasional paper). Sydney: Carrick Institute of Learning and Teaching in Higher Education.

Stark, J. (2000). Planning introductory college courses: Content, context and form. Instructional Science, 28(5), 413-438.

Stark, J., & Lowther, M. (1988). Strengthening the Ties That Bind: Integrating Undergraduate Liberal and Professional Study. Ann Arbor, MI: Professional Preparation Project.

Southwell, D., & Morgan, W. (2009). Leadership and the impact of academic staff development and leadership development on student learning outcomes in higher education: A review of the literature. Sydney: Australian Learning and Teaching Council.

Another spectrum for using indicators to place course websites

This post adds another perspective borrowed from Gonzalez (2009) as a framework to report or evaluate findings from Col and Ken’s indicators project. Col added an update on his work recently. Like previous post this one borrows a table of dimensions around conceptions of online learning because it may be helpful.

First the table and then how it might be used.

Dimensions

Dimensions delimiting approaches to online teaching – (Gonzalez, 2009: p311)
Informative/individual learning focuses Communicative/Networked learning focused
Intensity of use Small range on media and tools used to support learnign tasks and activities (mainly sources of information with small opportunities for interaction and communication) Wide range of media and tools used to support learning tasks and activities (with emphasis on interaction and communication)
Resources Web pages with information. Lecture notes. Links to websites. Web pages with information. Lecture notes. Links to web sites. Discussion boards. Chat. Blogs. Spaces for sharing. Animations. Videos. Still images.
Role of the learner Select and present information Design spaces for sharing and communication. Support the process.
Role of the students Study individually information provided Participate in a process of knowledge building

How might it be used

The above dimensions could be used to develop “analysis routines” that would place courses within these dimensions. Some potential approaches:

  • Variety and use of tools and media within a course site. (Intensity of use and Resources)
    Group the different tools available in the course management system into different types. e.g. those used for information distribution and those for interaction/communication. Count the number of different types of tools present in a course site and the level of usage.

    The difficulty here is the increasing use of non-CMS based tools for communication. e.g. I know of an increasing number of staff and students who are using external tools such as Messenger to work around the limitations of CMS services.

  • Measure student and staff activity (Role of the lecturer/students)
    I believe Blackboard, the main CMS at our institution, tracks the activity in some detail of each course site participant. If the type of activity can be categorised into groups (e.g. adding information to the site, using information on the site, posting to a discussion forum, responding to a post in a discussion forum etc.) then analysis could be run against the activity of all participants. This would identify the type of role the main groups are taking on.

What’s the value of this?

I can hear some thinking, “so what!”. What is the value of this sort of thing? A couple of thoughts.

  • As a framework to help make sense of the data.
    From my perspective it appears that the project is “drowning in data” and could use some sort of reviewed framework with which to organise or structure their investigations. These dimensions might provide it.
  • Enable institutions to get a handle on what is happening.
    Most of it ain’t great. The combination of the dimensions and the data potentially enable institutions, that are spending a lot of money on course management systems, to improve the awareness they have of what is actually happening. At the very least some sort of indication of where online courses site within the institution, as imperfect as it will be, sit within the dimensions might start some conversations about online practice that is actually somewhat informed by the reality of what is going on.
  • As a demonstration of building on the work of others.
    It is possible to argue with the value/validity of the knowledge generated by Gonzalez (2009) – but then it’s possible to argue against the validity of just about the knowledge generated by any research project depending on your perspective. However, this work is in a fairly prestigious journal, so it comes with a certain stamp of approval. This will help Col and Ken.
  • Perfect opportunity for a publication.
    Building on the last point, suggests that complimenting the qualitative nature of Gonzalez (2009) with some more quantitative measures from a broad collection of students and courses sounds like a pretty good publication opprotunity (or three).

It’s the potential for discussion within the organisation that is, I believe, of potentially the most beneficial for the most people.

The potential for publication is probably the most interesting to the project participants and frankly by far the easiest.

Further publications

The publication idea would be strengthened if previous work in this area (e.g. the recent ALTC project Learning and teaching performance indicators report either doesn’t do something like this or uses a different set of dimensions.

In addition, Gonzalez interviewed only 7 academics within a single discipline within a single institution. Chances are the results and dimesions identified in the paper are going to exhibit some sort of limitation, potentially caused by the nature of the context. Using a different approach in a different context will at least compliment/reinforce the findings and potentially identify additional dimensions.

References

Gonzalez, C. (2009). “Conceptions of, and approaches to, teaching online: a study of lecturers teaching postgraduate distance courses.” Higher Education 57(3): 299-314.

Data mining of online courses – dominant assumptions and innovation potential

For almost as long as learning management systems have been around their have been researchers and technologists investigating how the usage logs of these systems can be harnessed to inform and improve learning and teaching. For a little while I was sort of involved in a project that would look at some of this – interest has waned in line with organisational realignments.

A colleague, however, is continuing on with a project in this area. The following is an attempt to reflect about what little I know about the area and see if there is anything of interest to Col. In the following I am trying to identify the dominant assumptions underpinning this sort of work in order to see if there are any holes which might be ripe for exploitation.

Disclaimer: Until recently I haven’t paid much attention to the literature in this area and my recent reading was sparse, interrupted and incomplete. Hence the following could do with the insights of those with more knowledge of the literature to correct oversights, misunderstandings and naive assumptions.

What are we talking about

The common approach to this process uses automated evaluation of system logs and databases (e.g. Zorrila and Alvarez, 2008, Hung & Zhang, 2006; and Heathcote and Dawson, 2005) using some form of data mining or similar technology in order to provide additional information for teaching staff or Universities about the quality of the student experience. The typical form is take the usage logs and databases from a learning management system, put them into some sort of data warehouse system and generate reports for academics and management.

This type of work typically seems to differ on a few characteristics:

  • The technology/algorithms used to analyse the system logs and present the results.
  • The type of theory of learning or quality online presence that is used to “measure” the quality of the online course or the student experience.

I guess also there will be different communities within e-learning that take a different slant on this work. For example, I believe the intelligent tutorial system folk probably have a large amount of research and literature focused at analysing the student experience using data mining in order to make informed decisions about what to advise the student to do next.

Dominant assumptions and opportunity for innovation

Col is doing some of this work as part of a research project within his Masters study. So, I’m assuming he is keen to get into some innovation, to do something that hasn’t been done yet. So, in the following I’m trying to identify the dominant assumptions that, at least in my experience, seem to characterise this work. The idea is that over turning these assumptions might reveal an interesting line of work.

The assumptions that I’m thinking about (are there more?) include:

  • Limit and focus to the LMS logs.
    Analysis of LMS logs tells only a very small part of the story about students, staff and their interactions within learning and teaching. Even if the focus remains on the logs of computer systems, most students and staff will be leaving a fairly large trail of usage information through a range of other computer systems that could be combined with those in the LMS.

    This is especially important as the type of information gathered by mining LMS logs is limited by the nature and design of the LMS. For example, the Webfuse system (Jones and Gregor, 2006) has a design that by default allows open access to all resources. That is no requirement for users to login. Each system has its own design limits.

  • Kaplan’s law of instrument continued.
    Most of this work is done with similar tools. System logs and databases passed through statistics analysis and/or data mining technology to generate information. While useful, the tendency to focus just on these technologies has the potential to lead to flaws due to Kaplan’s law of instrument. Everything looks like a nail, if all you have is a hammer. Are there alternative tools or approaches that can be used, which may be base on different approaches and hence reveal different insight.
  • Emphasis on quantitative analysis alone.
    Simple log usage figures only tell a very small part of story of the student experience. Much of it is hidden within the words and emotions used and experienced by participants. It’s increasingly widely recognised that a multi-method approach for research is effective through each method covering limitations of other methods. Marrying quantitative analysis with textual analysis (e.g. Leximancer), or qualitative or specific feedback from students (e.g. course barometers; Jones, 2002) might be interesting.
  • Presenting information for staff and the institution, not the students.
    The aim for most seems to be to make the information available to the teaching staff so they can reflect and make improvements. Have seen little work where this information is made pro-actively available to the students so they can use it to guide their learning or student experience. For example, if a student were to see a huge spike in visitations to a particular page that they hadn’t visited, they may consider taking a look.
  • How is the insight generated used?
    Almost all of the research literature I’ve seen so far shows this information solely being used by researchers after the fact. i.e. after the term has finished they’ve analysed the data to see what lessons they can learn. The information may not even be shared with the teaching staff and they rarely seem to talk to academic staff. It would be interesting to see what would happen if you worked with academic staff during a term harnessing a range of different analysis/mining tools to provide information that they could respond to. This would allow tracking of what happens and the generation of insight and requirements within a real context of use, which is likely to be more useful and insightful than requirements gather out of context.

References

Heathcote, Elizabeth and Dawson, Shane (2005) Data Mining for Evaluation, Benchmarking and Reflective Practice in a LMS. In Proceedings E-Learn 2005: World conference on E-learning in corporate, government, healthcare & higher education, Vancouver, Canada.

David Jones, Student feedback, anonymity, observable change and course barometers, World Conference on Educational Multimedia, Hypermedia and Telecommunications, Denver, Colorado, June 2002, pp. 884-889.

Good teaching is not innate, it can be “learned” – and what’s wrong with academic staff development

The title to this post is included in a quote from Kane, Sandretto and Heath (2004)

The research team, comprising two teacher educators and an academic staff developer, embarked upon this research confident in the belief that good teaching is not innate, it can be learned. With this in mind, the project sought to theorise the attributes of excellent tertiary teachers and the relationships among those attributes, with the long-term goal of assisting novice academics in their development as teachers.

This coincides nicely with my current task and also with an idea I came across on the week-end about deliberate practice and the work of Anders Ericsson.

The combination of these “discoveries” is also providing some intellectual structure and support for the REACT idea about how to improve learning and teaching. However, it’s also highlighting some flaws in that idea. Though the flaws aren’t anywhere near as large as what passes for the majority of academic staff development around learning and teaching.

The following introduces these ideas and how these ideas might be used to improve academic staff development.

Excellent tertiary teaching

Kane et al (2004) close the introduction of their paper with

We propose that purposeful reflection on their teaching plays a key role in assisting our participants to integrate the dimensions of subject knowledge, skill, interpersonal relations, research/teaching nexus and personality into recognised teaching excellence. We conclude with a discussion of the implications of our model for staff development efforts.

Their proposition about the role of reflection in contributing to excellent teaching matches with my long held belief and perception that all of the best university teachers I’ve seen have been those that engage in on-going reflection about their teaching, keep looking for new knowledge and keep trying (and evaluating) innovations based on that knowledge in the hope to improve upon their teaching.

The authors summarise a long history of research into excellent teaching that focused on identifying the attributes of excellent teachers (e.g. well prepared, stimulate interest, show high expectations etc.) but they then suggest a very important distinction.

While these, and other studies, contribute to understanding the perceived attributes of excellent teachers, they have had limited influence on improving the practice of less experienced university teachers. Identifying the elements of “good” university teaching has not shed light on how university teachers develop these attributes.

The model the develop is shown below. The suggest

Reflection lies at the hub of our model and we propose that it is the process through which our participants integrate the various dimensions

Attributes of excellent tertiary teaching

The authors don’t claim this model to have identified any novel sets of attributes. But they do suggest that

the
way in which the participants think about and understand their own practice through purposeful reflection, that has led to their development of excellence

What’s been said about reflection?

The authors have a few paragraphs summarising what’s been said about reflection in connection to tertiary teaching, for example

Day (1999) wrote “it is generally agreed that reflection in, on and about practice is essential to building, maintaining and further developing the capactities of teachers to think and act professionally over the span of their careers”

.

They trace reflection back to Dewey and his definition

“an active, persistent, and careful consideration of any belief or supposed form of knowledge in light of the grounds supporting it and future considerations to which it tends

The also mention a framework of reflection outlined by Hatton and Smith (1995) and use it to provide evidence of reflection from their sample of excellent teachers.

Expertise and deliberate practice

Among the many quotes Kane et al (2004) provide supporting the importance of reflection is this one from Stenberg and Horvath (1995)

in the minds of many, the disposition toward reflection is central to expert teaching

Another good quote (Common 1989, p. 385).

“Master teachers are not born; they become. They become primarily by developing a habit of mind, a way of looking critically at the work they do; by developing the courage to recognize faults, and by struggling to improve”

Related to this view is the question “Was Mozart, and other child prodigies, brilliant because of some innate talent?”. This is a question that this blog post takes up. The answer it gives is no. Instead, it’s the amount and quality of practice they engage in which makes the difference. Nurture wins the “nature versus nurture” battle.

The blog post builds on the work of Anders Ericsson and the concept of “deliberate practice”. The abstract for Ericsson et al (1993) is

The theoretical framework presented in this article explains expert performance as the end result of individuals’ prolonged efforts to improve performance while negotiating motivational and external constraints. In most domains of expertise, individuals begin in their childhood a regimen of effortful activities (deliberate practice) designed to optimize improvement. Individual differences, even among elite performers, are closely related to assessed amounts of deliberate practice. Many characteristics once believed to reflect innate talent are actually the result of intense practice extended for a minimum of 10 years. Analysis of expert performance provides unique evidence on the potential and limits of extreme environmental adaptation and learning.

Implications for academic staff development

If reflection or deliberate practice are key to developing mastery or expertise, then how do approaches to academic staff development and associated policies, processes and structures around university learning and teaching help encourage and enable this practice?

Seminars and presentations probably help those that are keen to become aware of new ideas that may aid their deliberate practice. However, attendance at such events are minimal. Much of existing practice seems to provide some level of support to those, the minority, already engaging in deliberate practice around learning and teaching.

The majority seem to be able to get away without engaging like this. Perhaps there’s something here?

References

Common, D.L. (1989). ‘Master teachers in higher education: A matter of settings’, The Review of Higher Education 12(4), 375–387.

Hatton, N. and Smith, D. (1995). ‘Reflection in teacher education: Towards definition and implementation’, Teaching & Teacher Education 11(1), 33–49.

Kane, R., S. Sandretto, et al. (2004). “An investigation into excellent tertiary teaching: Emphasising reflective practice.” Higher Education 47(3): 283-310.

Sternberg, R. and Horvath, J. (1995). ‘A prototype view of expert teaching’, Educational Researcher 24(6), 9–17.

Somethings that are broken with evaluation of university teaching

This article from a Training industry magazine raises a number of issues, well known in the research literature, about the significant limitations that exist with the evaluation of university teaching.

Essentially the only type of evaluation done at most universities is what the article refers to as “level 1 smile sheets”. That is student evaluation forms that ask them to rank what they felt they learn, what they felt about the course and the teacher. As Will Thalheimer describes

Smile sheets (the feedback forms we give learners after learning events) are an almost inevitable practice for training programs throughout the workplace learning industry. Residing at Donald Kirkpatrick’s 1st level—the Reaction level—smile sheets offer some benefits and some difficulties.

His post goes on to list some problems, benefits and a potential improvement. Geoff Parkin shares his negative view on them.

The highlight for me from the Training mag article was

In some instances, there is not only a low correlation between Level I and subsequent levels of evaluation, but a negative one.

The emphasis on level 1 evaluation – why

Most interestingly, the article then asks the question, “why do so many training organisations, including universities, continue to rely on level 1 smile sheets?”

The answer it provides is that they are too scared to do them in case of what they find. It’s the ostrich approach of sticking the head in the sand.

What else should be done?

This google book search result offers some background on “level 1″ and talks about the other 3 levels. Another resource provides some insights and points to other resources. I’m sure if I dug further there would be a lot more information about alternatives.

Simply spreading the above findings amongst the folk at universities who rely and respond to findings of level 1 smile sheets might be a good start. Probably necessary to start moving beyond the status quo.

What do students find useful?

In a growing category of blog posts I’m expanding and attempting to apply my interest in diffusion theory and related theories to increase the use of course websites. A major requirement, as outlined in the previous post, in achieving this requires and understanding of what students find useful?

In this post, I’m trying to bring together some research that I’m aware of which seeks to answer this question by actually asking the students. If you know of any additional research, let me now.

Accessing the student voice

Accessing the student voice is the final report from a project which analysed 280,000 comments on Course Experience Questionnaire’s from 90,000 students in 14 Australian Universities. The final report has 142 pages (and is available from the web page). Obviously the following is a selective synopsis of an aspect of it.

The report summarises (pp. 7 and 8) the 12 sub-domains which attracted the highest percentage of mentions which it equates to those that are important to students. In rank order they are

  1. Course Design: learning methods (14.2% share of the 285,000 hits)
    There were 60 different methods identified as the best aspect of their studies which fell into 5 clusters

    • 16 face-to-face methods that focused on interactive rather than passive learning
    • 7 independent study and negotiated learning methods
    • 20 practice-oriented and problem-based learning methods
    • 6 methods associated with simulated environments and laboratory methods
    • 11 ICT enabled learning methods
  2. Staff: quality and attitude (10.8%)
  3. Staff: accessibility (8.2%)
  4. Course Design: flexibility & responsiveness (8.2%)
  5. Course Design: structure & expectations (6.7%)
  6. Course Design: practical theory links (5.9%)
  7. Course Design: relevance (5.6%)
  8. Staff: teaching skills (5.4%)
  9. Support: social affinity (3.8%)
  10. Outcomes: knowledge/skills (3.8%)
  11. Support: learning resources (3.5%)
  12. Support: infrastructure & learning environment (3.4%)

Going into totals

  • Course design – 40.6%
  • Staff – 24.4%
  • Support – 10.7%
  • Outcomes: knowledge/skills – 3.8%

Link to the 7 principles

A quote from the report

The analysis revealed that practice-oriented and interactive, face-to-face learning methods attracted by far the largest number of ‘best aspect’ comments.

Of the 7 Principles for Good Practice in Education mentioned in the last post, #3 is “encourages active learning”

What about CQU students

In late 2007 we asked CQU’s distance education students three questions

  1. What did you like or find useful?
  2. What caused you problems?
  3. What would you like to see?

Students were asked to post their answers to an anonymous discussion forum. This means they could see each others posts and respond.

An initial summary of the responses is available and CQU staff can actually view a copy of the discussion forum containing the original student comments.

A simple analysis revealed the following top 10 mentions

  1. 106 – Some form of eLecture – video, audio etc.
  2. 86 – Quick, effective and polite responses to study queries.
  3. 66 – Clear and consistent information about the expectations of the course and assignments e.g. making past assignments and exams available.
  4. 55 – Study guides.
  5. 53 – Good quality and fast feedback on assignments.
  6. 33 – For resources that are essentially print documents to be distributed as print documents.
  7. 30 – A responsive discussion board.
  8. 27 – Online submission and return of assignments.
  9. 25 – More information about exams, including more detailed information on how students went on exams.
  10. 21 – Having all material ready by the start of term.

CQU Students – 1996

Back in 1996 CQU staff undertook a range of focus groups of CQU distance education students aim at identifying issues related to improving distance education course quality. This work is described in more detail in a paper (Purnell, Cuskelly and Danaher, 1996) from the Journal of Distance Education.

Arising from this work were six interrelated areas of issues. These issues were used to group the suggested improvements from the DE students, these improvements are explained in detail in the paper and are summarised below.

  1. student contact with lecturers/tutors,
    • Easy access to people with relevant expert knowledge and skills (usually the lecturer).
    • Flexible hours for such access.
    • Some personal contacts through telephone and, where possible, some face-to-face contact.
    • Additional learning resources, such as audio- and videotapes to provide more of a personal touch.
  2. assessment tasks,
    • Detailed feedback (approximately one written page) on completed assessment tasks indicating how to improve achievement.
    • Timely feedback so that students can utilize feedback in future assessment tasks in the unit.
    • A one-page criteria and standards sheet showing specific criteria to be used in each assessment task and the standards associated with each criterion (statements of the achievement required for a high grade, etc.).
    • Clear advice on assessment tasks in the unit resource materials and in other contacts such as teleconferences.
    • Where possible, the provision of exemplar responses to similar assessment tasks be provided in the study materials.
    • Lecturers to be mindful of the differences in resources available to rural students compared to those in larger urban areas when setting and marking assessment tasks.
  3. flexibility,
    • Non-compulsory residential schools available at various locations of no more than three days’ duration and incorporating use of facilities such as libraries.
    • Greater consideration of the complexities of lives of distance education students by encouraging, for example, more self-paced learning.
    • Access to accredited study outside traditional semester times.
    • Lecturers/tutors to consider more fully the needs of isolated students in rural areas in support provided.
  4. study materials,
    • Ensure study materials arrive on time (preferably in the week prior to the commencement of a semester).
    • Efficient communications with students-particularly with the written materials provided in addition to the study materials.
    • Ensure each unit’s study guide matches other resources used in a unit, such as a textbook.
    • Lecturers should be mindful of extra costs for students to complete a unit in which, for example, specialized computer software might be needed; if a textbook must be purchased, it should be used sufficiently to justify its purchase.
    • Lecturers should cater to the range of students they have, especially from rural areas, with the study requirements for each unit (many participants reported that self-contained study materials in which there was little or no need to secure other resources to achieve high grades were valued).
  5. mentors, and
    • Having access to mentors is desirable but should be optional for students.
    • Issues about the role of a mentor need to be clarified.
  6. educational technology.
    • Continue to use and make more effective use of technologies familiar to students, such as the telephone and audio- and videocassettes.
    • Examine ways of minimizing access costs to the Internet for students, especially in rural areas.
    • Provide appropriate technical support for students to be able to access and use the Internet.
    • Provide professional development for staff to meet individual needs for using educational technologies involving, for example, interactive television, audio graphics, CD-ROM, e-mail, and the World Wide Web.

The commonalities between this list, from 1996, and the list generated in 2007 are not small.