BIM – getting student registration working

So, getting back into BIM development. The last post reminded me where I’m up to. The following is an attempt to plan, implement and document some code. Am starting where I left off, with the registration process for students. As part of this process I am finally starting to use the to do list for what it was meant to be used for.

Properly creating the form

The registration screen is being shown for student users. But it’s not with 100% “proper” Moodle code. i.e. it’s not using the forms library. The process is meant to be something like this:

  • Create a separate PHP class that has the detail for the form.
  • Work the code for normal usage/processing of the form into the appropriate PHP file.

I believe, for my needs, this translates into:

  • Create a PHP file in the module directory that follows the template.
  • Use that form in the view.php file.

Create the form

Found a slight difference. The template uses a form where it extends the class moodleform. The example I am using from the quiz module extends the class moodleform_mod. The _mod seems to suggest a special class for use in modules?

According to the xref docs on the forms stuff the moodleform class is a

Moodle specific wrapper that separates quickforms syntax from moodle code. You won’t directly use this class you should write a class definition which extends this class or a more specific subclass such a moodleform_mod for each form you want to display and/or process with formslib.

So it looks like some of the older docs are a bit out of date/wrong. But then docs in the same file suggest otherwise.

A page on moodleform_mod

At this stage, I’m again suffering my standard problem with Moodle. There seems to be no one place or one approach that helps you get an overview of how things fit together. This shouldn’t be this hard.

Moving forward

It was at this stage that I discovered a book that promises to give a more coherent overview of the task of creating an activity module. Initial impressions are okay. The true test will be in the work I describe below.

Using the moodleform class. I’ve been able to get the form being displayed. So, the next steps are:

  • Get it displaying what is required.
    The textual part of this is reasonably easy (I currently believe). Just using the addElement( 'html', $unregistered ) method to add some HTML in a variable. IMPORTANT: The HTML should not be hard coded in a variable. Needs to go in the lang files.

    All this “extra” work/abstraction in terms of form is necessary because of the nice way Moodle handles forms and auto puts it into the database (generally). This requires that the names of form elements match the field names in the database. Will also need to identify what other information needs to be held within the form in order update the database.

    The database table that holds the student registration is called bim_student_feeds and has the following fields

    • id – set automatically when values inserted NOT NEEDED
    • bim – the id of the particular instance of bim being referred to. Will need to use this when inserting. Already have this in view.php
    • userid – should be able to get this from existing structures, no need to be in the form.
    • numentries and lastpost – both can be null/0 initially.
    • blogurl – from the form
    • feedurl – needs to be calculated from the blog url
  • Figure out how to process that information.
    This seems to be quite straight forward. In the view.php there are a couple of if statements to control what is done if the user presses the save or cancel buttons. Have this control flow going already. Just need to add in a bit of checking and the code to check the student’s blog URL and extract the feed URL from the blog and consequently show appropriate error messages and/or insert the data.
  • Get the data entered in the form into the database.
    Quite straight forward using the Moodle database API calls. The biggest question here (and not a real big on) was to double check the “ids” that are being inserted/checked in the database.

Check IDs

The details about where each students’ blog is located, along with some additional information, is stored in the bim_student_feeds table. The fields of this table are listed above. There are three “ids” in the table:

  1. id – the unique id for each entry in table.
  2. bim – id for the specific bim activity, link back into the bim table. At the moment, I believe the assumption is that there can only be one bim activity per course. Will need to check this, probably no reason that there can’t be more than one (from a code perspective).
  3. userid – the id of the user/student for the feed.

To display the register form for students, there is a bim_feed_exists function that returns true if there is already a feed for the student. If there isn’t, display the register form, else display the details of the students form.

Obviously, the registration form code has to insert the appropriate data into the table. At the moment, there is a bit of a discrepancy about how that is working. Due mainly to the ad hoc way the data has been added. Time to fix that.

What should happen:

  • bim_feed_exists should be checking to see if there is already a combination of bim and userid in the table. id doesn't matter, as for duplicate feeds, it will be different. Must change that.
  • view.php needs to insert the right data. And here's the problem. Still hard-coded to 0.

What's done

At this stage the student registration process is working. If by working the definition is that the student without any registered blog URL will see the registration form. They can enter their URL and it will be inserted into the database. There is a list of tasks outstanding here:

  • Blog URL needs to be validated as correct.
  • Feed URL needs to be extracted from the blog url.
  • Some logging of registration needs to be added.
  • Appropriate error checking and reporting needs to happen with the process.
  • .. many more..

Where to now

Again I come up to the question of depth (complete all the necessary detail for registration to work properly) versus breadth (get all of the screens done so I can show folk).

At this stage, I'm leaning towards the breadth approach. I need to show the users of BIM what things are going to look like, even with the chance of seeing it in action. I face this question previously and chose depth, however, now that I've got database/form processing working I'm thinking the breadth approach might work better.

So, the plan is:

  • Design, create and populate the rest of the database tables for bim.
    This will allow some of the "show details" screens to have real data to draw upon. The data will be modified/anonymised versions of real data from BAM.
  • Work through the remaining screens so that they appear to be working but without many of the additional/real working they will need in the end.
  • Release these iteratively to the folk thinking of using BIM next year.

BIM and Moodle development – a more coherent overview found?

Today has been a fairly frustrating today with a mixture of organisational “stuff” and an increasing level of annoyance at the state of the public documentation around Moodle development slowing down BIM development. That state is essentially with stuff all over the place, no coherent path through it and regularly discrepancies between advice from different sources, or sometimes the same. But that’s the nature of documentation and open source projects.

The one bright spot today has been stumbling over this book which appears to fill the hole. Even if it is commercial. Having said that, the realease as writing nature of the book and some other aspects (donations to moodle) seems to indicate an enlightened company. On first skim the book looks good. Time will tell. Though, I am guessing that due to the nature of the type of book, complexity of the task and that it’s still being written there will be some rough edges (already submitted my first errata), or areas not covered.

Bugger, small problem. The one aspect I’m currently struggling with (use of moodleform_mod within a Module to create a new form – i.e. not in mod_form.php) is covered in their example. However, the authors have used moodleform and not moodleform_mod as the base/abstract class. I thought that was a no no?

Well, if I take their approach and use moodleform. It appears to get past the syntax errors being caused by my lack of understanding. Perhaps I just need to progress with that until future enlightenment.

So some progress today.

Here come the indicators, wait for the task corruption

Over the weekend it was reported in various media outlets that the Australian government has a new plan for higher education. A plan that includes words like “results-based funding”, performance, targets, quality and “readily available”. They cynical tone that I hope you are hearing, is not solely – or even mainly – due to some opposition to the idea of universities being accountable or effective. It’s mainly due to the belief that I think for the majority of teaching at universities this move will encourage more task corruption than it will actual real improvement in the quality of teaching.

This post argues for that perception. A bit further down (and in the comments), there is an argument against the practice of requiring new university teaching staff to complete teaching qualifications.

Do you agree? Disagree? Do you know of literature, blog posts or people that have argued otherwise? Are there strategies/advice that can be adopted to limit this sort of stuff (beyond what I’ve suggested at the end?)

Results-based funding: some detail

As outlined in this article there will be four broad areas:

  1. Student participation and inclusion.
  2. Student experience.
  3. Student achievement.
  4. Quality of learning outcomes.

The associated discussion paper goes into more detail.

Section 6 of the discussion paper lists the principles “developed to guide the choice of indicators for performance funding arrangements”. The list includes this one (my emphasis added)

be derived from high quality, objective data sources, and where possible collected at ‘arms length’ by an independent body, as well as not easily manipulated;

Part A of the discussion paper then offers more detail on each of the four measures/indicators. Including the provision of exact information about how the measure will be evaluated. The following information is given on page 14 for the Student Experience measure

  • Goal – improve student engagement and satisfaction.
  • Engagement
    • Target population – 1st year domestic under-graduate students.
    • How target will be articulated – Percentage point improvement in retention rate.
  • See the bit in bold in the above list? Yep, Percentage point improvement in retention rate. i.e. if you pass more first year students, you’ll do better. Retention is also used in the “Student achievement” indicator.

    Obviously pass rate is not a measure that can be “easily manipulated”?

    One perspective on pass rate

    A number of years ago, I was told that one part of one university was seriously considering paying casual teaching staff based on the number of their students who passed. It is my understanding that most people could see the problem with this. If people were paid based on students passing, then most – if not all – of the students would pass. Surely, this is an extreme example?

    Trouble is that it isn’t. Lot’s of evidence of this. The following is taken from Tutty et al (2008)

    More insidiously, there was evidence that rather than just inhibiting change, these
    ‘‘quality measures’’ may actually encourage inferior teaching approaches……The solution to the high failure rate was to change the assessment to satisfy the institutional requirements of satisfied students and reasonable pass rates rather than explore an alternative learning and teaching approach – an effective solution in the current higher education environment that encourages the academic to prioritise other areas, such as research.

    Even if it is not as straight forward as connecting payment to passing, the message in the above is that growing importance of certain quality measures will influence the behaviour of individuals. I believe this is essentially what Goodhart’s law suggestions

    any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes

    Oh dear, teaching qualifications again

    It’s with great sadness that I see the following included as a target for the “Quality of Learning Outcomes” indicator

    Agreed increase (n) in proportion of teaching only and teaching and research staff in academic organisation units with a Graduate Certificate in Higher Education or equivalent.

    The obvious assumption here is that if you have a graduate certificate in higher education (or equivalent) you will be a better teacher and your students will have better learning outcomes.

    The trouble with this is two-fold:

    1. The evidence for support of this causal linkage is weak (e.g. Stes et al, 2009).
    2. It ignores the impact of task corruption and Goodhart’s law.

    The first is more difficult to talk about giving perceived limitations in the research. Let’s assume that the connection does exist. What happens when institutions are being encouraged/required to increase the percentage of staff with Graduate Certificates in Higher Education? Goodhart’s law would seem to suggest that the causal connection/statistical regularity will collapse.

    i.e. the staff being forced to complete the Grad Cert will engage in compliance behaviours. They will do enough to get the Grad Cert with the likely outcome that they will not undergo any serious change in their understandings of L&T or in the behaviours. Consequently, student learning outcomes won’t change.

    Better solutions

    This section is included here because some folk complain that I am quick to point the flaws and slow to point to solutions.

    The broadest solution I’ve suggested is given in this presentation. It requires quite a significant mind shift from the current simplistic, quasi-corporate decision making being adopted around learning and teaching to an approach more informed by what we know about complex systems and the motivations and cognition of human beings.

    The simplest solution is embedded in this suggestion on how to improve outcomes on course experience questionnaires. An approach that essentially requires the identification of the smallest changes that can be made to bring courses into line with what we know about student preferences and ensuring that appropriate resources are allocated.

    Another solution I’ve suggested is the REACT process, especially if it were resources the right way. Done correctly, I think it can fulfill one of the main findings from a recent OECD report on teaching quality. i.e.

    Encouraging bottom-up initiatives from the faculty members, setting them in a propitious learning and teaching environment, providing effective support and stimulating reflection on the role of teaching in the learning process all contribute to the quality of teaching.

    References

    Tutty, J., J. Sheard, et al. (2008). “Teaching in the current higher education environment: perceptions of IT academics.” Computer Science Education 18(3): 171-185.

    Getting back into BIM: Summary and way forward

    The last couple of months have resulted in an absence from work on BIM (BAM into Moodle). This post is meant to be a summary of where I had gotten up to and a restatement of what I need to do. The latter part is somewhat uncertain due to limited communication within my local context. Somewhat disappointing.

    On the plus side, contributing to my lack of work on BIM was the attendance at a couple of conferences, including ASCILITE’09. It was obvious from a number of ASCILITE presentations that BAM/BIM remains an important and innovative tool that is much needed.

    If you’ve only started following this blog recently, this is they type of post acts as a log/diary of the work I have done on BIM. Probably not much of interest to a broader community.

    Current status

    Up until now most of the work has been familiarising myself with the “Moodle way”. I’d gotten to the stage where there was:

    • A BIM activity module in the Moodle source tree.
    • The module would create the BIM database tables.
    • The ability to add BIM as an activity to a course.
    • The resulting link inserted into the course would operate as the student details screen (screen dump) and show details about the students blog posts.
    • This screen would react to different data within the database.

    The one remaining, fairly important task to be done was to figure out how to get Moodle to take data from the user and appropriately put it into the database. This is essentially the one remaining component of web application development.

    Once familiar with this process, the implementation of BIM becomes a translation process from the previous system into Moodle.

    How it will work

    This is developed in other posts, but I’m going to re-create it here to refresh all those dormant memory patterns I have. Much of it is pretty similar to BAM. I’ve included links to screen dumps from BAM of most of the screens listed below. BIM screens will be based on these, but they will likely be different.

    The intent is that BIM will work in the following way:

    • Creation.
      • Course coordinator will add BIM as an activity to their course.
      • In doing so, the coordinator will need to fill in the configuration screen with details of the use of BIM (assignment name, posts etc.)
      • At the end of this process, there will be a link in the Moodle course to the BIM activity. This is used by both staff and students.
    • Normal use.
      • All users of BIM will use the link to the BIM activity added in the creation stage (e.g. this screen dump). Depending on what role they perform, they will see different information:
        • Students: there are two different views for students
          • Register Blog screen – appears when the student has not registered their blog. Shows some explanatory text and a text box. The student has to paste in the URL for their blog and hit submit. If there is a valid feed, success.
          • Student details screen – appears once the student has registered their blog. It will show what BIM knows about the student blog (where it is, the student’s name, the number of posts) and also any marking information (marks and comments) that have been released (only staff within an appropriate role can release marks/comments).
        • Coordinator: will have access to three different screens (all which are also used elsewhere by other people – with some modification).
          • Configuration screen – the same as the screen they see when adding BIM as an activity. Allows them to view and change the configuration of BIM for the course. There will need to be some limitations on what can be changed, but this needs to be thought through.
          • Show details screen – shows an overview of the details for all students. “All” students can be chosen as for the course, for a particular marker etc. Details include student name, number, number of entries in blog, and link to live blog.
          • Show posts screen – shows for “all” students an indication of whether they have posted answers to a particular question; number of posts blogged and marked; summary of marks etc.
          • Mark post screen – this is where the students’ post is marked (if required – not all courses using BIM mark blog posts).
          • Re-allocate screen – BIM attempts to automatically allocate student posts to the question (set by the coordinator) that it matches. This automated match doesn’t always work. This screen shows all of the student’s posts and allows the coordinator to change the allocation of a post.
        • Other staff: Will have access to the Show details screen, Show posts screen, Re-allocate screen and Mark post screen as described for the coordinator. The main difference is that “other staff” can only see details for students that they have been allocated to mark.
      • Aggregation, mirroring and allocation of blog posts.
        At a regular time interval (set at the system level) BIM will check if there is a new post to each students’ blog. If there is, BIM will
        • Add a copy of the new post(s) to a local RSS file, one per student.
        • Examine the new post(s) to see if they match any of the required questions as specified by the coordinator. If there is a match, the database will be updated to indicate another question has been answered.
        • Update the database that some new posts have been made.

    The Innovation Prevention Department: Why?

    The final keynote at ASCILITE’09 was by James Clay and was titled the Future of Learning (this is a link to an apparently earlier presentation by same author, same topic). Many aspects of the talk resonated with many in the audience, however, the one that perhaps resonated the most was that of the Innovation Prevention Department.

    James, as he describes in this comment was suggesting that most organisations have one department that seems to hold back innovation. The comment reveals James’ use came from Jon Trinder (slide 6). A quick google reveals the phrase being used as a chapter title in this 2002 book. So, it doesn’t seem to be a new concept that there always seems to be one department within an organisation that prevents innovation.

    So, what’s the problem?

    The problem is that in giving a list of departments that could possible fulfill this role, James started with the information technology department and that’s about where most of the audience seemed to stop listening. Many didn’t hear the other suggestions in James’ list. From where I was sitting, as soon as IT was mentioned most of the audience started nodding their head and remembering specific examples of where their IT folk had thwarted some innovation. It wasn’t long before “the Innovation Thwarting Department” play on information technology department was doing the rounds.

    As it happened, there were a couple of IT folk in the audience and another couple listening to the tweet stream. Not surprisingly, they were somewhat chagrined at this disparaging label for the work they do. Mark Smithers talks about his dismay at seeing the tweets from ASCILITE mentioning this. Nick Sharrat shares his thoughts about

    the frustration I often feel when my profession is disparaged for actually just ‘doing it’s job’, especially by people who often display an incredible naivity about the real world of IT.

    Not surprisingly, people don’t like being disparaged.

    Is there something there?

    Both Mark and Nick give lots of examples of the difficulties that IT face in doing their job. The constraints, which are many, within which they have to operate. They give examples of where the request or idea from the user is significantly flawed from a different perspective.

    However, isn’t it a worry when a significant percentage of the audience at a conference like ASCILITE’09, when presented with “innovation prevention department”, immediately though of the IT department? Rather than simply explaining why IT folk are rational, professionals working in a complex environment, shouldn’t there be an interest in understanding why the ASCILITE crowd are thinking this way?

    Given that ASCILITE is about computers in learning in tertiary education the folk at ASCILITE are keen to use computers effectively for that task. Given that IT professionals are be a key component/enabler of this work, shouldn’t they be getting on?

    I can’t see how the two groups can work together effectively if there exists this gulf in perceptions. If the source of the gulf is understood, perhaps that will enable the gulf to be bridged.

    Does anyone know of any work that has sought to document the causes for this gulf between IT and L&T folk? What about identifying strategies for moving forward?

    My suggested reasons

    Both Nick and Mark give a variety of reasons/problems for why/how IT function. The following is a start of some reasons that I propose for why L&T folk have formed the “innovation prevention” impression of IT. I’m doing this because I believe this is the first step in moving forward. Let’s have both sides get their cards on the table and then figure out ways forward.

    I must emphasise that the following list is based on my experiences, reading and perceptions. It is not meant to be definitive and may not describe what really happened, however, they do capture my perceptions of that reality. Please understand that the perceptions people have of what goes on is what drives how they act. It doesn’t matter whether or not you believe that the reality is otherwise. People will react based on their perceptions, not yours.

    I should note that I have an IT background. I’ve taught IT professionals. Some of whom have worked within IT departments in higher ed. I’ve also run a large scale IT system that was not part of central IT (though it is now).

    The assumption of objectiveness, rationality and professionalism.

    Nick argues that

    So, next time your IT department seems to be out to get you, give them a little more credit – you need to trust that they are proffesionals making very difficult compromises.

    Being a professional brings with it the aura of objectiveness and rationality. The trouble is that people are not information processing intelligences that make rational decisions. Our intelligence is based on pattern matching, our pattern matching processes are rife with biases and shortcomings. For example, the following was the finding reported by the technical team (consisting mostly of IT professionals) on the comparisons between different LMSs being considered at my institution in about 2003/2004

    strongly feels that the Blackboard product has the best overall technical fit and provides the best opportunity available to meet our tactical needs while minimizing support problems and costs

    . This is in spite of the observation that Blackboard had never been run on the existing infrastructure, one of the other LMS was a locally grown system that had been running on existing infrastructure for a number of years, and that a year or so after the implementation of Blackboard the institution had to invest in an entirely new server infrastructure due to problems in running Blackboard on the old infrastructure.

    And that’s before we get into politics. I’m sure any number of people within organisations can point to situations where the politics of the situation has driven the decision. IT departments are not absent of politics.

    Note: this does not mean that L&T folk are better, more rational, than IT folk. It’s just that both sets of people are prone to irrationality and biases.

    Who specifies the needs around innovation?

    Nick specifies the role of IT professionals as

    to provide systems that meet the business needs. That’s ‘needs’ and not ‘wants’

    . The trouble is that when it comes to innovation you can’t specify, you can’t plan. I use a quote from Joseph Gavin Jr in my email signature

    If a major project is truly innovative, you cannot possibly know its exact cost and its exact schedule at the beginning. And if in fact you do know the exact cost and the exact schedule, chances are that the technology is obsolete.

    It’s the emphasis on specification and planning that is in-built into most IT projects that is a direct anathema to innovation.

    Limited understanding of the nature of teaching and learning.

    This continues on from the previous point. IT is focused on specification, global solutions, the same solutions for all. Learning and teaching is all about diversity, variability and change. Features that do not match well with traditional IT processes. I’ve argues this in a recent presentation video (and slides).

    IT assume that the same processes they use for student records systems will work for learning and teaching.

    The user deficit model.

    Nick’s comment about people with “an incredible naivete about the real world of IT” in some IT folk (but by no means all and I don’t know Nick so this is not meant to be a characterisation of him) demonstrates a user deficit model. i.e. the users are stupid, we need to make the decisions for them, we know what’s best for them.

    This type of model is embodied in the acronym PEBKAC and it encourages a blame the user approach to thinking. Read the criticisms of PEBKAC to see how often user error/stupidity is due to the lack of quality in the IT systems.

    There’s a trite little saying

    There are only two industries that refer to their customers as users. The computer industry and the trade in illicit drugs.

    It may be trite but it shows a mindset that can and does exist in some IT folk.

    That said, there’s also a similar mindset towards/deficit model of teaching academics held by L&T support staff.

    The wrong rules

    While I was writing this, the following tweet came from Matthew Allen tweeted a point made by Skewes at the Broadband Futures event in Australia

    the problem is not technology; it’s the rules which prevent innovation

    . This ties in somewhat with the first point, but it’s also more than that.

    The vast majority of the practices of IT folk arose from a period when IT resources and the ability to use them were scarce and expensive. Increasingly with the advent of social media, the cloud, SaaS etc, I think we’re seeing the rise of a period of abundance in terms of the ability and availability of certain types of IT resource (some others may remain scarce).

    The rules for handling scarcity are the wrong rules for handling abundance.

    Change the environment, not the culture

    I’ve heard a number of folk at ASCILITE’09 claim that there needs to be a change in culture amongst academics around learning and teaching. To me this has always sounded a bit like a deficit model of teaching staff. It’s a model that I’ve heard again and again at ASCILITE’09 and in other literature around learning and teaching in higher education. You know the sort of thing e.g.

    Staff spend years getting their PhD in Physics and are then automatically expected to be good teachers.
    Most academic staff don’t have any qualifications in education…..

    i.e. there are problems with the staff and this needs to be fixed by innovation X (e.g. LAMS, learning designs, PLEs…), practice Y (e.g. all staff must have an education qualification) etc. We need to herd these problematic cats into a more productive direction.

    In this situation, culture is being used to describe the academics and what they do. It is ignoring the other component, which I’ll call the environment for the purposes of this post. My proposition is that it’s not the culture you want to change, it’s the environment. The environment is not conducive to the type of outcomes people want, and while the environment remains the same, no amount of changing the culture will have any effect at all.

    (As you’ll see below, some/most literature tends to use culture to refer to what I’m calling the environment. Sorry for the confusion, but I’m trying to engage with what some folk have been saying.)

    The following discussion may also connect with the cascading change symposium at ASCILITE and perhaps the idea that we need to focus on 3rd order change, not 2nd order change.

    The environment

    The university environment is not conducive to innovation, improvement or an emphasis on learning and teaching. Simply put, academics get rewarded for research. In terms of learning and teaching, they get words, but not action, about the importance of teaching for promotion and a collection of top down impositions and moves to standardisation. An environment conducive to compliance and corruption behaviour, not improvement in quality.

    For example (Twining et al, 2006: p 72)

    While the senior management team has an important role to play in fostering an ethos that supports change (see p77 Leadership), it is also clear that the wider educational context plays a vital role as well. At present the culture within education does not encourage people to take risks or innovate. Additionally, many educators in schools are still coming to terms with initiatives that they perceive have been imposed upon them. This can lead to ‘initiative blindness’ (Interview 26 – LA) which acts as a barrier to further change.

    Reflective alignment

    Academics are knowledge workers. Here’s a quote from Peter Drucker on knowledge workers

    The knowledge worker cannot be supervised closely or in detail. He can only be helped. But he must direct himself, and he must direct himself towards performance and contribution, that is, toward effectiveness.

    This quote is from the book “The effective executive” first published in 1967.

    I would describe the approaches being taken at my institution, and many others, as leaning much more towards the controlling/directing of the academic, rather than the helping. The increasing corporatisation of university are leading to policies, processes and systems that require standardised approaches across units and institutions, that move towards the controlling/directing rather than the helping.

    I’ve argued before that there are 3 observable levels of improving learning and teaching (re-purposing Biggs work on learning)

    1. What the teacher is.
      This is the old laissez faire approach to teaching in universities. There are some good teachers, there are some bad teachers. The institution isn’t really worried about this. Won’t try to do anything, except perhaps the really bad ones in out of the way courses.
    2. What the institution does.
      This is where most institutions are now. A university can’t have bad teachers. So to improve teaching and teachers the institution does lots of stuff. Institutional level policies, practices, studies, systems etc. Management take a big hand in developing and driving these things. They search for fads and fashions. Graduate attributes, LMS, e-portfolios, grad certs in L&T and any other centrally driven, institutional level approach tends (but not always) to fall into this category.

      Another symptom of this level of improving teaching is on-going restructures in the learning and teaching support groups (central and faculty). It’s indicative of senior managers arguing about who owns, controls and decides what the institution does. It’s indicative of who controls the agenda and attempts to control the academics and the culture that arises.

    3. What the teachers does.
      This is where we have to go (IMHO) and what I talk about below.

    What the teacher does?

    The simple argument (and yet very difficult to implement) is that the environment should encourage teaching staff to expend effective energy and effort on reflecting on their teaching and learning and acting on that teaching and learning. It should not focus on what the institution does. Some detail follows.

    In terms of improving learning and teaching, I follow this perspective

    Master teachers are not born; they become. They become primarily by developing a habit of mind, a way of looking critically at the work they do developing the courage to recognise faults, and struggling to improve. (Common, 1989)

    The reflection and struggle to improve has to be rewarded. This means that the vast majority, if not all, of the academics need to believe it will be rewarded and not just because senior management says it is.

    The institutional environment has to help academics reflect and struggle to change. This has to happen in a contextualised way. An individualised way. A way that breaks down disciplinary boundaries and group think. Top down doesn’t work. An approach that focuses on creating connections between people with the appropriate knowledge and capabilities to help the reflection and the struggle to improve. Generic global analysts and project managers do not help. The network of people has to include people with the combination of knowledge of learning, teaching, technology, the literature and the context. The network of people have to be encouraged to interact in regular and unexpected ways.

    The focus of the network has to be on helping the academic reflect and struggle to improve. Not on specifying de-contextualised best practice, but on helping reflection and improvement.

    Focusing on 3rd order change

    The cascading change symposium at ASCILITE was described in part as

    This symposium brings together a diverse and international group of researchers to explore the problems and limitations of using social media as a leverage point for second-order change in higher education. It aims to engage contributors and the audience in theoretical and empirical reflection on possible directions for further conceptual and methodological development in that area.

    I wondered whether we should focus on 3rd order change, not 2nd order change. Some definitions of “orders of change” – my re-phrasing of Bartunek et al (1987):

    • First order change – incremental changes which occur within an existing way of doing things or paradigm. i.e. you tweak at the edges without radically changing the context/culture/assumptions
    • Second order change – a fundamental change in the way of doing things, or the paradigm. i.e. you change the context/culture/assumptions to something entirely new.
    • Third order change – you develop the capacity of the system or the components of the system to change the context/culture/assumptions.

    I believe that the above suggestion, a focus on what the teacher does and especially focusing the environment on helping the academic to reflect and struggle to improve is more likely to create 3rd order change. i.e. the aim isn’t to help them to undergo 2nd order change, as that assumes you/they have decided what they need to change to. The focus of reflective alignment is on enabling 3rd order change.

    Small interventions, fundamental shifts

    James Clay wrote a blog post about the cascading change symposium. He makes the following observation

    My opinion is that these changes or interventions we make that we report at these conferences are always small and tiny and therefore can’t make a huge differences. We need to make major interventions at a institutional or even at a societal level if we are to effect fundamental change.

    I agree. Most of the interventions at a conference like ASCILITE are amongst the innovative teachers or from the support staff people who help them. We can’t make the type of institutional level change that I believe is necessary to achieve the 3rd level of improving learning and teaching. We can’t change the environment. In his post, James identifies one of the reasons

    People with the power to effect change do not (in the main) attend such conferences and therefore such changes do not happen at an institutional level.

    Until the top level folk engage in this type of thinking and move away from the 2nd level of improving learning and teaching, which perhaps could be described as a focus on 2nd order change, I don’t think there will be any significant change.

    Of course, one of the problems in getting management to engage is another aspect of the current environment. Short-term contracts for senior managers which contribute to them wishing to take charge, engage in large-scale projects of “2nd order change” (typically connected with the latest fad going through the community), move on and consequently trumpet the value of what they did before it becomes obvious there were no real change.

    Perhaps the ASCILITE conference organisers should actively invite a few institutional leaders to attend the conference – i.e. not do the welcome but sit in the sessions, engage in the discussions as yet another attendee.

    References

    Bartunek, J. and M. Moch (1987). “First-order, second-order and third-order change and organization development interventions: A cognitive approach.” The Journal of Applied Behavoral Science 23(4): 483-500.

    Common, D. (1989). “Master teachers in higher education: A matter of settings.” The Review of Higher Education 12(4): 375-387.

    Twining, P., R. Broadie, et al. (2006). Educational change and ICT: an exploration of priorities 2 and 3 of the DfES e-strategy in schools and colleges, Becta ICT Research: 106.

    Self-reporting considered harmful?

    I’m currently on my way to Auckland for the 2009 ASCILITE conference. A few weeks ago I was lucky enough to be in Denver for the 2009 EDUCAUSE conference. Both were/will be useful and interesting experience. But having been at EDUCAUSE, read a couple of the ASCILITE papers and talked with a colleague I’m once again wondering

    Should self-reporting be considered harmful?

    I’m now into the second day at ASCILITE, taken me that long to get back to this.

    Educational technology and self-reporting

    My interest and both conferences are interested in the application of educational technology within universities. A fairly typical, if not predominant, model of the work done in this area and presented at these conferences follows the following process:

    • Some one/group has a good idea and/or sees a problem.
    • They design a project and implement.
    • They evaluate the project.
    • They write and present the paper at a conference/journal.

    i.e. the people that had the idea and were responsible for the design and implementation of the intervention are the same people that evaluate and report on the success or otherwise of the project.

    They self-report.

    Why is this a problem?

    Is this really a problem? These folk should be professionals. They should be following accepted research methods, documenting those methods and having the results peer reviewed. Shouldn’t this address any problems with self-reporting?

    In a perfect world this might well be the case. But we are in far from a perfect world. I have a theoretical argument why this is the case and empirical observations to back it up.

    The theory arises from science and is the simple observation that human beings are not rational decision makers. We are not information processing machines (i.e. the brain is not a digital computer). We are pattern matching intelligences (anyone familiar with Dave Snowden will have heard this before). We look for and see/interpret situations through the lens of first-fit pattern analysis. Patterns formed by regular and recent experience get hit first.

    This theory/fact is probably a strong contributor factor to the Kaplan’s Law of Instrument.

    My problem is that, in the best case, the person designing the intervention will already have an established set of expectations/patterns that they are looking for. They may not test for information outside of those expectations. e.g. an straight information technology intervention will be measured on whether it delivered the promised features on time and on budget and not by the implications on the users and how they work around the limitations of the system and the resulting costs.

    At worst, you will get post-hoc rationalisation of the sort talked about in relation to planning in this post “Make it looked planned”. Where order that didn’t exist during the design, is added on afterwards.

    I can point to at least two papers at the ASCILITE’09 conference that suffer from this problem. Rationale and order that was not present in the design and implementation have been added into the paper after the fact. I’m pretty sure that these aren’t the only two papers.

    Now, in some cases, a bit of post hoc rationalisation may be a good thing. Especially where the implementation of the project is itself intended to be a learning process. Anything truly innovative can’t be planned. What I’m complaining about here is when certain principles or goals are added after the fact and the impression given that they were present from the start.

    Solutions?

    This is incredibly difficult to do in any widespread way.

    One solution that we played with a little bit, but which never got off the ground was the REACT process. In this process, the problem, possible solutions, the chosen solution and implementation plans were presented and peer reviewed before implementation. This not only provides some evidence that post-hoc rationalisation hasn’t occured, it also provides an opportunity for good people to provide additional input and ideas into a project before it is implemented. Hopefully strengthening the proposal.

    After this, the intervention is implemented and evaluated and then reported back.

    I wonder if you could get something like this going around a conference? Perhaps we should try and get it going back at CQUniversity first.

    Barriers to harnessing academic analytics

    The Indicators project is an attempt to enable the examination, analysis and comparison of LMS usage across time, systems and institutions. The project is nothing new. Projects around academic analytics have been around for a while and I and others at my current institution have been talking about using the collective data in systems logs to inform the practice of L&T at universities for a long time.

    Given the long term interest and the need to do this, why aren’t more universities doing it? What is getting in the way? This post is the start of a list of factors that I have gotten an inkling of through the literature, personal experience and talking with others. What would you add?

    (PS. I’m trying to follow advice in this post on improving a blog post)

    Nature of the technology: Limitations of the LMS

    Currently, for most universities, the learning management system (LMS – Blackboard, Moodle etc) are the main environment for L&T. For good or bad, this is a major source of data. Some of the barriers are inherent to the nature of the LMS. These include:

    • Minor variations in naming and approach.
      Even though there are more similarities than differences between different LMS (Black et al, 2007), those differences are enough to make comparisons between LMS somewhat difficult. There needs to be work like that carried out by Malikowski et al (2007) to enable meaningful comparisons.
    • The LMS focus on individual courses.
      An LMS is focused on enabling individual academics create and maintain course sites. This means the features, including reporting, are targeted at the course level.
    • The LMS doesn’t encompass all data.
      To really understand the impact of LMS activity you need to be able to match student activity with student performance (i.e. their grades). Typically, student grade information is not available within the LMS. If you all rely on is the LMS data, you can’t access this and other information (e.g. demographic information about students or staff).
    • Advice to clear system logs.
      The versioni of Blackboard my institution used to run came with some advice to purge the activity accumulator. One of the major tables/systems of the LMS that recorded what was being done with Blackboard. Purging it means destorying the data. Unless the IT folk involved stored the data first, you lost it.
    • The general poor quality of the reporting/visualisation features of an LMS.
      From Dawson and McWilliam (2008) “the current LMS present poor data aggregation and similarly poor visualisation tools in terms of assisting staff in understanding the recorded patterns of student learning behaviour.”

    Technical furphies

    It’s been my experience that when technical people don’t really want to something, for whatever reason, they come up with a furphy to explain why it can’t be done. The best of these furphies have an element of truth in order to make them plausible. However, if you know a bit more about technology, there are usually simple solutions. The ones I’m aware of include:

    • It will overload the live database.
      The idea here is that trawling the LMS database to generate analysis of how it is being used will raise “concerns with system load for querying the database in order to extract the required data” (Dawson & McWilliam, 2008). i.e. the users of the LMS will suffer from poor performance. The problem with this furphy is that any IT department worth its salt will already have a copy of the database that is being regularly updated that it uses for development and testing. If they don’t, they’re not doing their job properly. That test database is not directly used by users and can generally be used without performance concerns. In addition, with any of the “enterprise” database systems, it shouldn’t be difficult to create another copy of the database, if they don’t want the analytics playing with the dev database.
    • We don’t have enough resources.
      This is perhaps a superset of the previous point. It usually means that we don’t have the database adminstrators, developers or hardware resources available to support the project. Providing access to an existing dev database on an “enterprise database” should take seconds – at least if the processes and automation used by the IT department is any good.

      In defence of the IT folk and also to make clear that these are not universals. In some cases, they don’t have the resources. Many institutions don’t put enough resourcing into IT to achieve what they want. In these cases, IT are probably better off collaborating with people who want to do things as a better way to generate support to get the additional resources.

    • The database is to complicated to understand.
      Some of the folk in L&T working in analytics don’t have technical backgrounds and must rely on IT. As another version of the previous points there is the argument that the databases are too complicated to easily get useful data from them.
    • It’s not us, it’s the system owner.
      We’d love to give you access, but first you have to talk to Y as they are the system owner. We’re pretty sure that they are not keen on the idea. Not sure how you might contact them. Just a bit of passive resistance.

    Organisational barriers

    • Ownership of the data.
      Some teaching staff see all information about their course in the LMS as belonging to them. Some faculties believe the same for their faculty courses. IT may see ownership of the database as belonging to them. The institution may not have any firm written policy on this and simply rely on who speaks most strongly. This problem often connected with the following.
    • Mismatch with system owner requirements.
      For example, the organisational owner of student records data at our institution has been the student administration section. Folk responsible for getting students enrolled, results processed and accepting money. These folk are not mostly focused on improving learning and teaching. So may not see the rationale for analytics or other tasks.
    • Privacy concerns.
      Fears that information about individual students or even staff, will be made public and or misused — leading to the next point.
    • Plans and fears of using the data as a stick.
      It’s not hard to see some management using reports from the LMS to identify “bad” academics and punish them. This may be achieved simply through the creation of KPIs and standards for course delivery that assume that it’s possible to make such universal statements and ignore the inherent variability in university learning and teaching. This leads to the next one….
    • Task corruption and Goodhart’s law.
      If you set a particular measure as a performance indicator/target (e.g. the presence of a discussion forum or the number of contributions to the discussion forum by the staff member) you will get people achieving that target. However, in some/many cases they will be using task corruption to achieve it.

    Solutions

    Many people complain that I point out problems with out pointing out solutions, hence the inclusion of this section

    The solutions we’ve used, mostly by luck than forethought, have included:

    • Historical responsibilities.
      At one stage, members of our project were system owners or designers of at least two of the LMS used by our institution. This meant that we have access to the databases for these systems. Though it didn’t help with the LMS we aren’t responsible for.
    • Accidents.
      As mentioned above, advice from Blackboard is that you regularly clear/rotate the activity accumulator. For some reason, our IT folk never did this. So all that data, going back to 2005, was available.
    • Technical knowledge.
      At least two of our project members have some technical knowledge. When faced with technical furphies we could point out alternate representations.
    • Organisational knowledge.
      I’ve been at my current institution for nearly 20 years (god that’s sad). For good and bad, I know a fair bit about the organisation and the people within it. Often it is who you know, not what. For example, we got access to student records data over 10 years ago because I knew the system owner of the student data regularly had lunch at a certain place. I just happened to be there to ask nicely to have access to the data.
    • Personal connections.
      Both internally and externally project members know a broad cross-section of people who can help provide alternate representations of stories that have been told. Representations that often unlock doors.
    • Organisational power.
      This is perhaps the most useful one. If you are doing work that is seen to be useful or important by one person in power, or lots of people throughout the organisation, that can often provide the necessary power to override/workaround objections.
    • Workarounds.
      Often you don’t have to ask permission. Often there are technical and social solutions to access that can work around barriers. For example, Bakharia et al (2009) talk about an approach to analyse course discussion forums that use Greasemonkey to avoid the need to access the LMS database.

    Of course, in a perfect world, all members of the organisation would be warm and fuzzy folk who are only too happy to collaborate with their colleagues to achieve important outcomes for the organisation. None of them would ever be advised not to provide assistance or support to individuals within the same organisation.

    References

    Bakharia, A., E. Heathcote, et al. (2009). Social networks adapting pedagogical practice: SNAPP. Same places, different spaces. Proceedings ascilite Auckland 2009. Auckland: 49-51.

    Black, E., D. Beck, et al. (2007). “The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments.” Tech Trends 51(2): 35-39.

    Dawson, S. and E. McWilliam (2008). Investigating the application of IT generated data as an indicators of learning and teaching performance. Melbourne, Australian Learning and Teaching Council: 45.

    Malikowski, S., M. Thompson, et al. (2007). “A model for research into course management systems: bridging technology and learning theory.” Journal of Educational Computing Research 36(2): 149-173.

    What research groups are looking at the future of education

    As outlined very briefly in a recent post I am involved in the early days of project that may result in some sort of proposal for a research group/centre around the future of education. This is meant to be some documentation of my early research to look for groups that are already doing work in this area.

    This is still very early days. There’s a strong chance that the “direction” will be refined and/or changed. For example, it might become concerned with the future of education within regional areas. It might not. Finding out what others are doing in the area, will help inform any changes.

    At the moment, I’m thinking of the following main groups to look for:

    1. Australian university research groups.
      By the nature this idea has to be aware of other groups and be seen to fit/collaborate with them.
    2. EC research groups.
      I have a vague recollection of the EU funding groups/research in this sort of thing.
    3. More broadly.
      In recent months the future of education, especially universities has been getting a bit of airplay within the media. There will be other folk doing work. Obviously there are also university/research groups from outside the EU and Australia. For example, if I google “future of education” it returns 69M+ hits.

    Apart from reporting the findings when I search the above, the following also includes “factors”. A collection of ideas/thoughts/factors people think either are or will influence the future of education.

    If you’re not interested in the filtering and summarisation I give below. You can view the list of resources I look at (but may not include below) on del.icio.us.

    This isn’t complete. A bit more to do.

    Australian research groups and resources

    Australian universities – academic education groups :

    Some fairly recent, short term insights into where various Australian governments are going.

    Australian governments review of australian higher education and future directions

    EU research groups

    All things education and training in the European Commission seems to be able to be found from here.

    More broadly

    Factors

    The following list of terms encompass factors which people think will influence the future of education. They include: openness, Web 2.0, “the flat world”,

    The map of future forces affecting education from the KnowledgeWorks Foundation provides a much more detailed outline, including the ability to dive deeper.

    This future of education post identifies a range of issues, but I’m somewhat underwhelmed by the conclusions.

    Conference with range of speakers from 2007 on future of education

    What am I doing?

    In the University within which I work there has always been one particular divide. There are the faculties – these are the parts of the organisation in which the “normal” academics live – and there are the non-faculties – populated by those folk who are not “normal” teaching/research academics sitting within disciplines. For example, I’m employed as an academic, I do research, but I do no direct teaching and live within the Curriculum Design and Development Unit. I’m supposed to help the institution improve it’s learning and teaching.

    The trouble is that almost all the academics are within the faculties. Those of outside the faculties are the “others” by being outside but we’re also somewhat invisible. In particular, what we do is typically seen as invisible because we aren’t within the faculties. Just recently someone I’ve known for sometime, asked someone else I work with, what I was doing. Coming from others, the question might have been, why are we still paying him if he’s not doing anything?

    So, this post is to make public what I’m doing for the institution and also (and mostly) for me to outline what I have to do.

    50/50

    At the moment, my work effort/resource is split 50/50 between my official role as the e-learning and innovation specialist for the institution, and in a very much related role with the Learning and Teaching Education Research Centre at the same institution.

    In my spare time, I’m trying to finish the write up of my thesis

    Elearning and Innovation Specialist

    In this position, I’m working on the following projects:

    • Support of people using BAM including support of existing courses/students/staff and helping new staff consider its use.
    • BAM into Moodle (BIM).
    • The indicators project.
    • Occasionally, some early discussions around curriculum mapping.

    A large part of this role, apart from these tasks, is reading material and presenting

    LTERC

    I’m helping LTERC staff in the formulation of an application for a new collaborative research centre for CQU around learning, teaching and education.