Call for participation: Getting the real stories of LMS evaluations?

The following is a call for participation from folk interesting in writing a paper or two that will tell some real stories arising from LMS evaluations.

Alternatively, if you are aware of some existing research or publications along these lines, please let me know.

LMSs and their evaluation

I think it’s safe to say that the idea of a Learning Management System (LMS) – aka Course Management System (CMS), Virtual Learning Environment (VLE) – is now just about the universal solution to e-learning for institutions of higher education. A couple of quotes to support that proposition

The almost universal approach to the adoption of e-learning at universities has been the implementation of Learning Management Systems (LMS) such as Blackboard, WebCT, Moodle or Sakai (Jones and Muldoon 2007).

LMS have become perhaps the most widely used educational technologies within universities, behind only the Internet and common office software (West, Waddoups et al. 2006).

Harrington, Gordon et al (2004) suggest that higher education has seen no other innovation result in such rapid and widespread use as the LMS. Moodle or Sakai. Almost every university is planning to make use of an LMS (Salmon, 2005).

The speed with which the LMS strategy has spread through universities is surprising (West, Waddoups, & Graham, 2006).

Even more surprising is the almost universal adoption of just two commercial LMSes, both now owned by the same company, by Australia’s 39 universities, a sector which has traditionally aimed for diversity and innovation (Coates, James, & Baldwin, 2005).

Oblinger and Kidwell (2000) comment that the movement by universities to online learning was to some extent based on an almost herd-like mentality.

I also believe that increasingly most universities are going to be on their 2nd or perhaps 3rd LMS. My current institution could be said to be on its 3rd enterprise LMS. Each time there is a need for a change, the organisation has to do an evaluation of the available LMS and select one. This is not a simple task. So it’s not surprising to see a growing collection of LMS evaluations and associated literature being made available and shared. Last month, Mark Smithers and the readers of his blog did a good job of collecting links to many of these openly available evaluations through a blog post and comments.

LMS evaluations, rationality and objectivity

The assumption is that LMS evaluations are performed in a rational and objective way. That the organisation is demonstrating its rationality by objectively evaluating each available LMS and making informed decisions about which is most appropriate for it.

In the last 10 years I’ve been able to observe, participate and hear stories about numerous LMS evaluations from a diverse collection of institutions. When no-one is listening, many of those stories turn to the unspoken limitations of such evaluations. They share the inherent biases of participants, the cognitive limitations and the outright manipulations that . Stories that rarely, if ever, see the light of day in research publications. In addition, there is a lot of literature from various fields suggesting that such selection processes are often not all that rational. A colleague of mine did his PhD thesis (Jamieson, 2007) looking at these sorts of issues.

Generally, at least in my experience, when the story of an institutional LMS evaluation process is told, it is told by the people who ran the evaluation (e.g. Sturgess and Nouwens, 2004). There is nothing inherently wrong with such folk writing papers. The knowledge embodied in their papers is, generally, worthwhile. My worry is that if these are the only folk writing papers, then there will be a growing hole in the knowledge about such evaluations within the literature. The set of perspectives and stories being told about LMS evaluations will not be complete.

The proposal

For years, some colleagues and I have regularly told ourselves that we should write some papers about the real stories behind various LMS evaluations. However, we could never do it because most of our stories only came from a small set (often n=1) of institutions. The stories and the people involved could be identified simply by association. Such identification may not always be beneficial to the long-term career aspirations of the authors. There is also various problems that arise from a small sample size.

Are you interested in helping solve these problems and contribute to the knowledge about LMS evaluations (and perhaps long term use)?

How might it work?

There are any number of approaches I can think of, which one works best might depend on who (or anyone) responds to this. If there’s interest, we can figure it out from there.

References

Coates, H., R. James, et al. (2005). “A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning.” Tertiary Education and Management 11(1): 19-36.

Harrington, C., S. Gordon, et al. (2004). “Course Management System Utilization and Implications for Practice: A National Survey of Department Chairpersons.” Online Journal of Distance Learning Administration 7(4).

Jamieson, B. (2007). Information systems decision making: factors affecting decision makers and outcomes. Faculty of Business and Informatics. Rockhampton, Central Queensland University. PhD.

Jones, D. and N. Muldoon (2007). The teleological reason why ICTs limit choice for university learners and learning. ICT: Providing choices for learners and learning. Proceedings ASCILITE Singapore 2007, Singapore.

Oblinger, D. and J. Kidwell (2000). “Distance learning: Are we being realistic?” EDUCAUSE Review 35(3): 30-39.

Salmon, G. (2005). “Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions.” ALT-J, Research in Learning Technology 13(3): 201-218.

Sturgess, P. and F. Nouwens (2004). “Evaluation of online learning management systems.” Turkish Online Journal of Distance Education 5(3).

West, R., G. Waddoups, et al. (2006). “Understanding the experience of instructors as they adopt a course management system.” Educational Technology Research and Development.

LTERC, finally a research centre – shamless plug

Sorry, but the purpose of this post is entirely selfish. My host institution has recently established a research center around learning, teaching and education. Given my background and interests, I’ll be doing work under the auspices of the center and occasionally need to access the website (even though the website is implemented with CQU’s home-grown content management system) of the center.

The full name of the center is the Learning and Teaching Education Research Centre which is abbreviated to LTERC. As of yet, despite promises to the contrary, you can’t Google LTERC and get a pointer to the site. To make matters worse the home-grown content management system doesn’t maintain human readable URLs. The URL for the centre is http://www.cqu.edu.au/lterc/, but once you’re looking at the page you this meaningful URL gets replaced with http://content.cqu.edu.au/FCWViewer/view.do?site=779. Obviously a URL that encourages one to remember the site. For example, I’ve only just (through trial and error) realised that the http://www.cqu.edu.au/lterc URL will work.

Yes, I know I could book mark it. I could even be innovative and use del.icio.us. But that doesn’t help make it easy to tell external folk about the site. Rather than use URLs, these days I tend to say “google it”. For example, rather than give the URL for the video of my talk last week (au.video.yahoo.com/watch/6075473/15784044) I can say google “herding cats losing weight”.

The point is ‘google “lterc”‘ doesn’t work yet. And it’s not as if there a massive amounts of people using “lterc” – it’s not like “david jones”.

So the point of this blog post is to get a few pages on the web pointing to the LTERC website so that Google might rank the site a bit higher.

Is this mis-use of the blog?

Using Votapedia

In the next couple of weeks I’m going to be giving a presentation that will also serve as an experiment in alternate technologies for presentations. One of those technologies will be Votapedia.com – an Australian-based, free SMS/Web audience response system. This post is meant to capture the process I went through in learning about how to use Votapedia.

Accounts

Votapedia is based on Mediawiki. To create quizzes on Votapedia you need to get an authorised account. This consists of two steps:

  1. create an account;
    Using the normal mediawiki approach.
  2. get the account authorised or known.
    This is entails sending an email to a person with some blurb about what you’re using it for. I got a response in a few hours.

Participants using the service to “vote” don’t need accounts. However, those voting via the Web can create accounts to use in voting – they don’t need to.

There are limits placed on how many times participants can “vote”, but this is done on the basis of the IP address (if via the web) or their phone number (via phone).

Creating a survey

Surveys exist on their own page on the Votapedia installation of MediaWiki. You can create surveys by using some specific markup or using one of a number of “forms” which automate the process. Let’s create one.

A part of the Votapedia home page is shown in the following image. The links in the first left-hand menu are how you create surveys. There are 6 different types of surveys.

Survey type Description
Simple survey
Questionnaire This is a survey with more than one question. Wit h this type of survey you don’t need to wait for everyone to finish a question before moving on.
Quiz Essentially a questionnaire, but with other features (allocate points, can’t see the quiz page before it starts..) to allow it to be used for student assessment.
Anonymous text Participants submit whatever they want
Identified text
Rank expositions

Votapedia home page

Each link takes you to a basic HTML form that guides you in the information required to create the chosen survey. The following image is for the simple survey (click on it to see more).

Creating a simple survey on votapedia

Well, that’s not good. It didn’t work. Filled out the form, all good, hit submit and I get “There is currently no text in this page”.

So, I try to create another survey. Very simple and don’t do anything to upset the gods. Same error. Not good. Go looking and see there’s a link “My Surveys”, perhaps that might give me the link. Yep, the two surveys do show up on that page, see the following image.

My Surveys on Votapedia

Okay, if I click on the link for one of the surveys on “My Surveys” page I get a page with the same “error”. Now, there is a “Choose Number” link for each survey, maybe I need to select that first.

It appears that Votapedia has a limited set of phone numbers, choose number means selecting from that collection of numbers, one number for each response. Some are in red – which means you can’t use them – some are in green.

Trouble is that you can only chose one number at a time and it always asks for me to choose the first number. What about the others?

It would seem that I am missing something important.

Tried to create an “anonymous text” survey, same problem. There are other surveys that seem to be working….

Mmmm, now they are working. The main thing that changed was that I changed the password for my account. I don’t think that will have changed anything. Here’s the proof.

My first Votapedia quiz

Still only able to choose one phone number. Well, let’s try and start the survey. Hit the “Start survey” button….that seems to start it. The numbers are already there. Let’s try. Phone the number for my response, and hey presto it works. Engaged tone and the graph is updated in front of my eyes. That’s neat. Time to tell some other folk.

Ahh, I’ve now got an SMS from Votapedia thanking me for my vote and giving me some details to login to the website.

Now I did want to try and change the survey while it was running. I wanted to remove the results from the page and enable web voting. But it didn’t look like I could change it while the survey was running. So I’ve turned it off and will reset the survey and see if the changes work.

But of course, this was because I was viewing the survey through my account. Not what a visitor would see. Silly David.

Running a survey

Just briefly, asked a few colleagues to take the survey – all up 7 participants. The experience highlighted:

  • Getting the engaged signal when dialing the number gave the impression of failure – this needs to be made clear at the start.
  • The web interface for taking the poll suffered the same problem that I described above when creating the question.

Results looked like the following image. Interestingly, it seems at least some of the participants missed the least modifier in the question.

Results of Votapedia question

Conclusions

Still a bit dodgy in places via the web interface. Phone side worked well. Will need the right sort of preparation of participants.

The question of the phone numbers and how long it takes to dial a response is also an issue.

Thoughts on “Insidious pedagogy”

The following is a reflection on and response to a paper by Lisa Lane (2009) in First Monday titled “Insidious pedagogy: How course management systems impact teaching”. I’ve been struggling with keeping up with reading, but this topics is closely connected to my thesis and the presentation I’ll be giving soon.

The post starts with my thoughts and reactions to the paper and has a summary of the paper at the end.

My Thoughts

In summary, the paper basically seems to be based on

  • observing a problem; and
    In summary, the problem is that because most academics are not expert online technology users they seek to use course management systems (CMS) at a basic level by using system defaults. They system defaults in some CMS (e.g. Blackboard) are seen to encourage limited use and also to encourage academic staff to continue as novices. These novice staff produce learning environments that are less than appropriate, but they are also happy with the CMS.
  • proposing two bits of a solution.
    The two solutions are:
    • start novices with pedagogy;
      When introducing a CMS to technically novice acacdemic staff, don’t start by examining the technical features of the CMS. This encourages them to stick with those features without considering pedagogy. Instead, start with pedagogy and work to the tools.
    • have the CMS use opt-in, rather than opt-out.
      The default setting for an opt-out CMS is that all of the options are there, in the face of the academic. This can be confronting and lead novices to taking more pragmatic approaches. An opt-in approach has fewer defaults which encourages/requires the academic to think more holistically.

I like the paper, especially in its description of the problem. This is an important problem that is often over-looked. However, while there is some value in the solutions – the distinction between opt-in and opt-out is especially interesting – I wonder about the practicality of the “start with pedagogy” solution. Also, not surpisingly given such a complex problem, think there are other factors to be considered.

Practicality of “start with pedagogy”

My current institution is currently in the midst of adopting Moodle. The institution has implemented the organisationally rational approach of having compulsory training sessions in Moodle being run by both IT trainers and curriculum designers. For various reasons, a number of the staff attending these sessions have asked a common question: “What’s the minimum I need to know?”. Such staff aren’t that interested in starting with pedagogy.

This raises an interesting point that I haven’t thought of for the first time. Given our institutional context, I believe that the number of true novices (i.e. those that have never used a CMS) amongst academic staff is very low. Many of these staff may well have very limited conceptions of e-learning from a pedagogical perspective, however, they have started to develop “their way” of teaching online. They are comfortable with that and all they want to know is how to replicate it in the new system.

In addition to this, most of the staff I know don’t start with pedagogy when they are designing their teaching. This can be due to not knowing about pedagogy or also for vary pragmatic reasons. For example, if you are a casual, part-time being employed to teach an existing course, you are going to stick with what has been done before. You’re not being paid to do something different and any problems that arise because of “difference” will not be treated well.

Other solutions

There are many other potential solutions, I will be talking about the main ones in a couple of weeks. Some other misc ones before I get back to work:

  • Engage web novice academics in the use of the Internet – especially social media – that further their career.
    e.g. Using social media to connect with other researchers, using blogs to become a “public intellectual”. This provides them with experience to be aware of different possibilities.
  • Modify the context of most universities to appropriately encourage a focus on improving learning and teaching.
    Are instructors motivated to spend more time on improving their teaching? What if they believe the following (Fairweather, 2005)

    More time teaching is a negative influence on academic pay….The trend is worsening most rapidly in institutions whose central mission focuses on teaching and learning

    Until universities truly value learning and teaching and treat it as such…….

  • Adopt a best of breed approach for the CMS.

Other thoughts

Other thoughts/responses include

Is Moodle really different?

Lane (2008) writes (emphasis added)

This is particularly true of integrated systems (such as Blackboard/WebCT), but is also a factor in some of the newer, more constructivist systems (Moodle).

This seems to accept the view that Moodle, being designed on social constructivst principles, is somehow different and better than Blackboard, WebCT etc. I’m sorry to say that if I haven’t seen anything significant while using Moodle that strongly shows those social constructivist principles.

I think there’s a really interesting research project around investigating this claim, how/if it is visible in the design of Moodle and how that claimed strength influences use of Moodle.

Today’s CMS can be customized

There’s a quote in the paper (Lane, 2008)

Today’s CMSs can be customized, changed and adapted

I question this a little. I think the point of the quote in the paper is from the perspective of the academic. i.e. that when designing your course there is choice, an ability to customize your course in a variety of ways by the breadth of additional functionality that CMS vendors have provided.

I agree with that to some extent, however, this customization has some limits:

  • don’t break the model;
    All systems have an in-built model. You can only customize to the extent that you fit within the model. We had an experience in one course where we couldn’t create enough discussion forums in the right places for one pedagogical design. This was entirely due to the assumptions built into the CMS about how discussion forums would work. It broke the model, so we couldn’t do it.
  • your installation allows it.
    There is an important distinction to be made between what the CMS allows you to customize and what the particular installation of the CMS you are using allows you to customize. The decisions made by specific institutions can further constrain the level of customization. The simplest example is the choice at the institutional level not to install “module X”. But in some CMS there are also installation level configuration decisions that constrain customization.

I’ve argued elsewhere that the basic model of a CMS is based on that of an integrated, enterprise system – a product model well known to be inflexible. In fact, best practice information systems literature suggests that for such systems you must “implement vanilla” to minimise costs.

Designed to focus on instructor effeciency?

The paper (Lane, 2008) includes the following claim about the design of CMSs

Today’s enterprise–scale systems were created to manage traditional teaching tasks as if they were business processes. They were originally designed to focus on instructor efficiency for administrative functions such as grade posting, test creation, and enrollment management.

My position is that most of them were very badly designed to do this, if they were at all designed to do this. I’ve heard lots of folk explain that if you have a class for 30 or 40, then the commercial CMSs work fine, but if you have 800, you are buggered.

The first version of WebCT installed at my institution had an internal limit on the number of students that could be managed within the gradebook – 999. If you had more than 1000 students in a class, you were stuffed. My institution had classes that big.

The nature of my current institution – courses having upwards of 20 different teaching staff spread across the eastern coast of Australia – means that online assignment submission and management is an important task. Experience of staff here is that the assignment submission system in Blackboard is really bad in terms of efficiency. Early indications are that the default Moodle system is just as bad. A locally produced system is significantly more efficient.

All of this seems to bring into question the “efficiency” aspect of CMS. They don’t even do that well. We should write something on this.

Paper summary

The following is a quick summary of the paper

Introduction

Nice quote from Thoreau which I might have to steal

But lo! men have become the tools of their tools.

Draws on historian’s view to argue that technologies tend to have a purpose/objective that can limit or even determine its use.

Course Management Systems (CMS) also do this, through the defaults in those systems. Other literature tends to not to focus on this. The paper suggests that

A closer look at how course management systems work, combined with an understanding of how novices use technology, provides a clearer view of the manner in which a CMS may not only influence, but control, instructional approaches.

The inherent pedagogies of CMSs

CMS designed mostly for administrative purposes. Built-in pedagogy is essentially based on presentation and assessment. The design of these systems make it simple to perform presentation and assessment tasks.

That said, CMSs have been expanded to include other features and this is expanding. Suggests that CMSs can be customized, changed and adapted. But why aren’t faculty tinkering the CMS to make their individual pedagogies work online?

Novice web users and the CMS

Most academics are not web-heads. Most are drafted to teach online. It’s based on top-down directives. Lots more references to explain that they aren’t savvy with technology. At the same time, most have established successful learnig approaches over time.

Interesting points about how much academics use the same research methods they learned in graduate school. Can expand here.

Experts and novices are different.

The fault of the defaults

Basically argues that the defaults of the CMS aren’t designed to make it easy for or fit with the expectations/experience of academics. As they spend more time with the system, they become comfortable with the defaults.

Important: makes the point about the difference of perspective between educational technologists and academics, especially how they view the CMS.

Novices are happier with CMS because – to put it bluntly – they don’t know better. It’s the folk pushing the boundaries that are less satisfied with CMS.

Solutions to CMS dominance

Treat novices, differently from advanced instructors. With novices emphasis pedagogy first. Argues that starting with technology features focuses on the novice instructor’s weakness (technological literarcy) at the expense of their main strength (expertise in discipline and their teaching).

Also suggests that “opt-out” systems – systems that show all the tools and features and expect users to choose which they don’t want – are too overwhelming for novices. Suggests that opt-in systems – like Moodle – are better. Especially in the way they give similar emphasis to discussions and content transmission

References

Fairweather, J. (2005). “Beyond the rhetoric: Trends in the relative value of teaching and research in faculty salaries.” Journal of Higher Education 76(4): 401-422.

Lane, L. (2009). “Insidious pedagogy: How course management systems impact teaching.” First Monday 14(10).

Lectures and the LMS: Alternatives and experiments

This post stores information about an experiment/presentation seeking to examine alternatives for both the lecture and the LMS. Information available below includes:

Resources

The video of the talk is available on ustream. Slides are below.

When

The following experiment/presentation will take place on Tuesday the 10th of November from 1pm-3pm. The time is based in the “Australia – Queensland – Brisbane” timezone, you can use this converter to make it meaningful for you or the following table might help.

Where When (start time)
London Nov 10, 3am
Washington DC Oct 9, 11pm
New Delhi Nov 10, 8:30am

What

The experiment/presentation will occur in two forms:

  1. Physically on a number of rooms on the campus of CQUniversity; and
    Rockhampton – 33/G.14. Bundaberg – 1/1.12. Gladstone – MHB 1.09. Mackay – 1/1.01.

    IMPORTANT: Originally the Mackay room was not going to be available. Due to the change in time the Mackay room is now available.

  2. Virtually through ustream, twitter and Votapedia.
    The ustream will probably be through this channel. More details on the twitter and votapedia usage will be given during the presentation.

The current session structure will be

  • Introduction and background – no more than 30 minutes.
    Explain the rationale for the experiment and get people using Votapedia and twitter.
  • Presentation – 50 minutes.
    A dry run of the EDCAUSE’09 conference presentation
  • Discussion and questions.
    Whatever time left will be for discussion amongst the participants.

Abstract

Postman’s (1998) fifth of five things to know about technological change is that media or technology tends to become mythic. That is, some technologies come to be thought of as part of the natural order of things. It becomes difficult to imagine life without the technology. Postman suggests that this is dangerous because such technology becomes accepted as is and is consequently not easily modified or changed. Such difficulty is a contributing factor to what Truex et al (1998) label as stable systems drag, where an organisation battles against its constraining technologies as it seeks to adapt to an ever-changing environment. There can be no doubt that universities operate in a continuously changing environment (CQU, 2005)

This session consists of a talk and an experiment. Both aim to explore and open up for modification two mythic technologies within higher education: the lecture and the learning management system. The talk will argue for the need for alternatives to learning management systems and describe the implementation and results of such an alternative. The experiment will use various technologies (ustream, Votapedia and Twitter) to demonstrate methods to significantly modify the mythic attributes of lectures and presentations.

You will be able to participate in the talk and the experiment either by coming to one of the ISL rooms on campus or by your web browser. If you do participate, please be sure to bring your mobile phone. If you’re really keen, you may also wish to create yourself an account on Twitter for use during the presentation.

Additional Background

The talk will be a trial run of a presentation to be given at EDUCAUSE’09 in early November. The title is “Alternatives for the institutional implementation of e-learning: Lessons from 12 years of Webfuse”. The abstract for the talk follows.

The practice of e-learning in universities suffers from a number of unquestioned perspectives that limit outcomes. This presentation describes a framework for understanding the full diversity of alternate perspectives and examines one successful set of perspectives arising out of 12+ years of designing, supporting and competing with the Webfuse system.

An extended abstract of the talk is also available.

The talk will be used as the test bed for an experiment with a range of different technologies that seek to question many of the mythic attributes of the lecture or presentation. The technologies being experimented with include:

  • ustream – a live interactive video broadcast platform.
    ustream provides a free, simple to implement and easy to use approach that allows anyone with a web browser to watch the presentation live.
  • Votapedia – a web and SMS audience response system (clickers)
    Votapedia allows the presenter to pose questions and poll participants answers during a presentation. Votapedia will allow anyone with a mobile phone of web access to participate in these questions and answers.
  • A back channel.
    Using a combination of Twitter and features of ustream participants will be able to share a conversation about the presentation while it is happening.

References

CQU. (2005). “CQU Strategic Plan: 2006-2011 – Creating an opening to a different future.” Retrieved 31 October, 2005, from http://policy.cqu.edu.au/Policy/policy.jsp;policyid=607.

Postman, N. (1998). Five things we need to know about technological change. NewTech, Denver, CO.

Truex, D., R. Baskerville, et al. (1999). “Growing systems in emergent organizations.” Communications of the ACM 42(8): 117-123.

Lessons from Pedagogy for e-learning

Two thesis related posts in a day, I must be on a roll. This post actually marks a milestone, the following rough bit of material is the last bit of original writing I’ll need to do for chapter 2. What remains will be tidying up, fixing typos/spelling/grammar, “concludings” and some major cutting. Sadly chapter 2 currently stands at 200+ pages and will need some major cutting I think to be a reasonable size. That’s a job for another day.

The following is meant to abstract some lessons for e-learning based on the literature around pedagogy reviewed in early sections (e.g. the one from earlier today. It continues my focus on diversity and change being key characteristics of e-learning, an observation that highlights a mismatch with the standard product and process being used for e-learning.

As I near the end there are an increasing number of cross references from this material to earlier material. Sorry, haven’t gotten around to linking them on the blog. This is likely to be only somewhat less annoying than the poor grammar and dyslexic typing.

Lessons from Pedagogy for e-learning

The above brief overview of the Pedagogy component of the Ps Framework forms the basis for the identification of four lessons for e-learning within universities from the literature on pedagogy. The first of these is that learning is an inherently diverse human activity. The second is that e-learning is only a relatively new human activity and is still changing and adding to the diversity of learning. The third lesson, and one based on this observation of increasing diversity, is that there is no silver bullet, no one universal approach to learning or to e-learning and that instead e-learning should perhaps be focusing on its ability to support this diversity. The final lesson is that any change in learning and teaching at university is reliant on changing the conceptions of the academics.

Learning is inherently diverse

Dede (2008) raises the question of whether or not there is just one pre-eminent way of learning/teaching for every student, for every subject, for all legitimate purposes of schooling? Like everything else in education, a balance is needed – one size does not fit all – even in online settings (Cuthrell and Lyon 2007). Different learners bring to the learning experience: different learning objectives; different prior knowledge and past experience; and, different cognitive preferences (Dagger, Conlan et al. 2005). The diversification and massification of the student body has led universities to shift their education rhetoric from a notion of “one size fits all” to a concept of tailored, flexible learning (Lewis, Marginson et al. 2005). Learning should not be one size fits all and can be customised to meet local requirements and this deviation from a standard model should now be seen as a strength (Cavallo 2004). A “one size fits all” approach ignores the importance of disciplinary culture (Jones 2009). There is no one best way of developing instruction (Davies 1991). Dede’s (2008) answer to his question is that given the spectrum of learning theories, it would appear that “learning is a human activity quite diverse in its manifestations from person to person”. He goes onto suggest that the field of instructional design can only progress if it recognises that learning is a human activity quite diverse in its manifestations from person to person and even from day to day (Dede 2008).

E-learning is new and changing

While, to some extent, Bates (2004) statement that e-learning does not change the fundamental process of learning in that students still need to read, observe, think, discuss, practice and receive feedback. However, e-learning is creating a new environment within which learning and teaching operates and is contributing to the creation of and need for new knowledge about learning and teaching. There is little understanding of the affordances of different technologies and how these might be exploited in particular learning and teaching contexts (Conole and Dyke 2004). There is a need to engage with the affordances and constraints of particular technologies to understand how new technologies can meet specific pedagogical goals of specific content areas (Mishra and Koehler 2006). The rise of e-learning is calling for and generating more than knowledge simply to inform instructional design theories. With the example of connectivism, it is possible to see new knowledge, enabled or required to some extent by the rise of technology, being generated at the other three levels of learning theories identified in Section 2.1.2.

E-learning, diversity and silver bullets

The diversity inherent in learning is not matched by the theories and philosophies around the use of information and communication technologies to support learning. Such approaches treat learning as a simple activity that is relatively invariant across people, subject areas and educational objectives; and, so most widely used instructional technology applications have less variety in approach than a low-end fast-food restaurant (Dede 2008). The apparent high costs of developing educational materials means, that at least for for-profit organizations, a “one size fits all” approach produces economies of scale that is likely to prevail over the potential of online technologies to support customisation for the needs of individual learners (Cunningham, Ryan et al. 2000). This tendency towards one size fits all is contributed to by successive generations of pundits espousing ‘magical’ media, the single best medium for learning or the universally optimal way of learning (Dede 2008).

The difference and diversity inherent in learning challenges managerialism – a rising trend within higher education as shown in Society in Place (cross reference) – which generally seeks to elide ambiguities and to standardise individuals and experiences (Danaher, Luck et al. 2004). The managerialist approach to standardisation is well served by the monolithic or integrated product model on which learning management systems are based (cross reference to procurement and software section in Product). Innovation and diversity are served less well by such a product model. Dede (2008) argues that

from an instrumental perspective, the history of tool making shows that the best strategy is to have simultaneously available a variety of specialized tools, rather than a single device that attempts to accomplish everything.

Improvement comes through changing teacher conceptions

Even with the diversity in learning and the change created by the introduction of e-learning, the practice of learning and teaching in universities remains much the same. While e-learning has provided a new medium, must teaching remains old wine in new bottles (Bates 2004). As shown in section 2.1.4 (e-learning usage from past experience) the majority of academic staff still rely on old, familiar pedagogies rather than actively engaging with the new affordances offered by technology. This is something that is only going to change when the university context encourages, enables and perhaps even requires, changes in the conceptions of learning and teaching held by academic staff. The on-going introduction of new technologies is unlikely to ever bring about such change.

References

Bates, T. (2004). The promise and myths of e-learning in post-secondary education. The Network Society: A Cross-cultural Perspective. M. Castells. Cheltenham, UK, Edward Elgar: 271-292.

Cavallo, D. (2004). "Models of growth – Towards fundamental change in learning environments." BT Technology Journal 22(4): 96-112.

Conole, G. and M. Dyke (2004). "What are the affordances of information and communication technologies?" ALT-J, Research in Learning Technology 12(2): 113-124.

Cunningham, S., Y. Ryan, et al. (2000). The Business of Borderless Education. Canberra, Department of Education, Training and Youth Affairs: 328.

Cuthrell, K. and A. Lyon (2007). "Instructional strategies: What do online students prefer?" Journal of Online Learning and Teaching 3(4).

Dagger, D., O. Conlan, et al. (2005). Fundamental requirements of personalised eLearning development environments. World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2005, Vancouver, Canada, AACE.

Danaher, P. A., J. Luck, et al. (2004). Course management systems: Innovation versus managerialism. Research Proceedings of the 11th Association for Learning Technology Conference (ALT-C 2004), University of Exeter, Devon, England, Association for Learning Technology.

Davies, I. (1991). Instructional development as an art: One of the three faces of ID. Paradigms rgained: the uses of illuminative, semiotic, and post-modern criticism as modes of inquiry in educational technology: a book of readings. D. Hlynka and J. Belland, Educational Technology Publications: 93-106.

Dede, C. (2008). Theoretical perspectives influencing the use of information technology in teaching and learning. International Handbook of Information Technology in Primary and Secondary Education. J. Voogt and G. Knezek. New York, Springer: 43-59.

Jones, A. (2009). "Redisciplining generic attributes: the disciplinary context in focus." Studies in Higher Education 34(1): 85-100.

Lewis, T., S. Marginson, et al. (2005). "The network university? Technology, culture and organisational complexity in contemporary higher education." Higher Education Quarterly 59(1): 56-75.

Mishra, P. and M. Koehler (2006). "Technological pedagogical content knowledge: A framework for teacher knowledge." Teachers College Record 108(6): 1017-1054.

Learning theories and e-learning

It’s almost a month since the last post from the first draft of my thesis. So, after much time away here’s the next installment. It’s probably rougher than previous versions – which says something – I’m still getting back into the swing of things.

The following is meant to be a description of learning theory within the context of e-learning at universities. It’s not a complete or in-depth examination of learning theories. Instead, it tries to illustrate that what we know about learning theory (in the broadest possible definition) is hugely complicated, diverse and ever changing. The intent is to argue that this is in direct contrast to characteristics of the common approach taken by universities to support e-learning. That is, an approach that focuses on stability and inflexibility.

Learning theories

The previous section (Section 2.8.1) argued that the quality of student learning within a university context is heavily influenced by the thinking, planning and strategies adopted by the pedagogues responsible for individual courses. This section seeks to summarise the research, literature and theories that have arisen to guide the thinking of pedagogues around learning. The following section cannot do justice to complexity, diversity, breadth and depth of research into learning. Explanatory accounts of learning range across culture, biology and cognition providing a multitude of theoretical perspectives drawing on different methodological traditions and bringing different educational phenomena into focus (Bell 2004). The scientific literature on cognition, learning, development, culture and the brain are voluminous (Bransford, Brown et al. 2000). Education, like other branches of the social sciences, has no single, unifying mature theory, instead theories, ideas and approaches coexist in various states of cohesion and tension (Dillon and Ahlberg 2006). There are many schools of thought on learning, and no one school is used exclusively to design e-learning (Ally 2004).

Given the diversity of perspectives, methodologies and schools of research associated with a variety of perspectives of learning, it is beyond the scope of this thesis to give a complete accounting of the research around learning. Instead the aim of this section is to establish the diversity, complexity, uncertainty and contradictions inherent in this research as it applies to the practice of e-learning within universities. This starts with a description the four levels of learning “theory” before a brief discussion of technology and learning theory.

The four levels of learning “theory”

Given the diversity of disciplinary and theoretical perspectives related to learning even defining learning and learning theory can be difficult. Definitions of learning differ based on approach and intended purpose and often reveal more about the perspective from which the person offering the definition sees learning (Siemens 2006). Definitions of what a learning theory is will likely differ between psychologists, computer scientists, instructional designers and other disciplines. However, it is possible to extract from literature such as Ertmer and Newby (1993) four levels or perspectives on learning “theory” or research. These four levels are: epistemology, descriptive theories from science, learning theories, and instructional design theories. Each of these four levels is briefly examined below with particular emphasis on the diversity, complexity and ever-changing nature of views within each.

Epistemology is concerned with the nature of knowledge and how we come to know things (Driscoll 1994), what does it mean to know (Siemens 2006). Ertmer and Newby (1993) in examining the connection between epistemology and learning theory identify two fundamental perspectives of epistemology: empiricism – the view that experience is the primary source of knowledge – and rationalism – the view that knowledge derives from reason without the aid of the senses. Performing a similar task, Driscoll (2000) adds a third epistemological perspectives of nativism – the belief that knowledge is innate or present at birth. Pallas (2001) identifies the proliferation of epistemologies as one of the most confusing developments in educational research over the past quarter-century and goes on to list a “welter of names” – positivism, naturalism, postpositivism, empiricism, relativism, feminist standpoint epistemology, foundationalism, and postmodernism.

Descriptive theories from science arise from disciplines including, but not limited to, the various branches of psychology, neuroscience, and biology that seek to understand how various aspects of human learning function. Seidel, Perencevich and Allyson (2005) argue that psychology can provide descriptive laws that describe how cognitive development, learning, meta-cognition and other elements of learning actually occur. Driscoll (1994) illustrates the influence of disciplinary perspectives by illustrating how behavioural, cognitive and social psychologists develop different views of learning. The contribution of theories arising from science is not limited to learning theory. Goldman (1986) argues that an understanding of the architecture of the human mind-brain is essential for primary epistemology. He continues to argue that epistemology, the history of which has shown strong currents against being interdisciplinary, should be a multidisciplinary affair (Goldman 1986).

Learning theories seek to provide insight into the act of learning (Siemens 2006). A learning theory comprises a set of constructs linking observed changes in performance with what is though to bring about those changes (Driscoll 1994). Discussions of different learning theories (e.g. Ertmer and Newby 1993; Driscoll 1994) tend to focus on three distinct viewpoints: behaviourism, cognitivism and constructivism. These learning theories link closely to the three different perspectives of behavioural, cognitive and social psychologists mentioned in the previous paragraph. Historically it can be seen that, the cognitive perspective overthrew behaviourism in the 1950s and 1960s and also underwent a major shift in the 1980s and 1990s towards constructivism (Mayer 1996).

The on-going historical development of learning theories has not stopped. Mayer (1996) suggests that a fourth metaphor is likely with possibilities arising from either an assimilation and accommodation between the existing metaphors, or from an entirely new approach. One such entirely new approach may be provided by connectivism, a theory describing how learning happens in a digital age (Siemens 2005; Siemens 2006) based on the epistemological foundation of connective knowledge (Downes 2006). Table 2.3 provides a summary of the three existing broadly accepted learning theories and connectivism.

Table 2.3 – Learning theories (adapted from Siemens, 2006)
Property Behaviourism Cognitivism Constructivism Connectivism
How does learning occur? Black box—observable behaviour main focus Structured, computational Social, meaning created by each learner (personal) Distributed within a network, social, technologically enhanced, recognizing and interpreting patterns
Influencing factors Nature of reward, punishment, stimuli Existing schema, previous experiences Engagement, participation, social, cultural Diversity of network
What is the role of memory? Memory is the hardwiring of repeated experiences—where reward and punishment are most influential Encoding, storage, retrieval Prior knowledge remixed to current context Adaptive patterns, representative of current state, existing in networks
How does transfer occur? Stimulus, response Duplicating knowledge constructs of “knower” Socialization Connecting to (adding) nodes
Types of learning best explained Task-based learning Reasoning, clear objectives, problem solving Social, vague (“ill defined”) Complex learning, rapid changing core, diverse knowledge sources

Table 2.3 does not capture the full diversity of learning theories. Mayer (1996) describes the six versions of constructivism identified by Steffe and Gale (1995) as “social constructivism, radical constructivism, social constructionism, information-processing constructivism, cybernetic systems, and sociocultural approaches”. Further, when the three main theories are closely analysed it becomes apparent that there are many overlaps in the ideas and principles (Ally 2004). Classifications of learning theories and theorists are contradictory (Siemens 2006) and confusing due to the use of different labels for categories, the grouping of major models and theorists in different categories and the use of vague concepts. Identifying where within the basic learning paradigms a particular theorists fits can be confusing due to theorists and their ideas evolving over time and subsequent changes to their ideas (Sackney and Mergel 2007). Rather than three competing theories, these can be though of as a taxonomony of learning with behaviourism being used to teach the “what”, cognitivism the “how” and constructivism the “why” (Ertmer and Newby 1993).

Instructional design theories are prescriptive theories that offer explicit guidance on how to better help people learn and develop (Reigeluth 1999). The origins of formal instructional design procedures have been traced back to the development of military training materials during the Second World War (Reiser 2001). Interest grew strongly during the 1970s and 1980s with a large increase in the number of instructional design models (Reiser 2001). Perhaps forming the basis for Postman (1995) claiming that while educators were once famous for providing reasons for learning, they had now become famous for inventing a method. Leading to a situation where the initial impression of these theories is one of diversity, followed by being perplexed by so many theories being at odds with one another (Duchastel 1998).

As a source of further complication, Shulman (1986) introduces the idea of pedagogical content knowledge (PCK) that argues that treating a pedagogue’s content knowledge and the pedagogue’s knowledge of pedagogy as mutually exclusive domains resulted in teacher education that emphasised one over the other. PCK is the blending of knowledge about content and pedagogy into an understanding of how particular aspects of content knowledge are best organised, adapted and represented within instruction (Mishra and Koehler 2006). Treating this knowledge as separate is not sufficient for capturing the knowledge good teachers require (Shulman 1986). Numerous research studies such that no optimal pedagogy is effective regardless of the subject matter (Dede 2008).

The above has sought to illustrate that “learning theory” consists of theoretical perspectives from at least four different levels. Each of those levels is characterised by significant diversity and in some cases confusion. In addition, some of the levels impact upon other levels in unexpected ways. The next section briefly discusses what happens when technology is added to pedagogy.

Technology and learning theory

There exist many different conceptual frameworks for describing the relationships among learning theories, pedagogical strategies, instructional designs and information and communication technologies (Dede 2008). E-learning, in general, does not change the fundamental process of learning (Bates 2004). However, research into how people learn online is in its infancy and further research is needed to provide insight into how to develop engaging and effective online learning environments in higher education (Herrington, Reeves et al. 2005). Writing in 2004, Bates (2004) suggested that since the use of the web for learning and teaching is less than ten years old, its application of learning and teaching was still evolving.

What research had been done suggested that the three established learning theories (behaviourism, cognitivism and constructivism) all provide principles that can be used to design online instruction (Ally 2004). Any given pedagogical tool may incorporate perspectives from any of these three intellectual positions (Dede 2008). The actual applications of e-learning are highly dependent on the teacher’s epistemological preferences and their chosen pedagogy (Bates 2004).

In extending Shulman’s (1986) work on pedagogical content knowledge (PCK) into the concept of technological pedagogical content knowledge (TPACK), Mischa and Koehler (2006) argue that

there is no single technological solution that applies for every teacher, every course, or every view of teaching”. Quality teaching requires developing a nuanced understanding of the complex relationships between technology, content, and pedagogy, and using this understanding to develop appropriate, context-specific strategies and representations. Productive technology integration in teaching needs to consider all three issues not in isolation, but rather within the complex relationships in the system defined by the three key elements.

Dede (2008) makes a similar point that no application of technology to learning and teaching is universally good. Instead the best approach is to analyse the nature of the curriculum, students, and teachers in order to select the appropriate tools, applications, media and environments (Dede 2008).

References

Ally, M. (2004). Foundations of Educational Theory for Online Learning. Theory and Practice of Online Learning. T. Anderson and F. Elloumi. Athabasca, Canada, Athabasca University: 3-31.

Bates, T. (2004). The promise and myths of e-learning in post-secondary education. The Network Society: A Cross-cultural Perspective. M. Castells. Cheltenham, UK, Edward Elgar: 271-292.

Bell, P. (2004). "On the theoretical breadth of design-based research in education." Educational Psychologist 39(4): 243-253.

Bransford, J., A. Brown, et al. (2000). How people learn: brain, mind, experience, and school. Washington, D.C., National Academy Press.

Dede, C. (2008). Theoretical perspectives influencing the use of information technology in teaching and learning. International Handbook of Information Technology in Primary and Secondary Education. J. Voogt and G. Knezek. New York, Springer: 43-59.

Dillon, P. and M. Ahlberg (2006). "Integrativism as a theoretical and organisational framework for e-learning and practitioner research." Technology, Pedagogy and Education 15(1): 7-30.

Downes, S. (2006, 3rd October 2009). "Learning networks and connective knowledge." Instructional Technology Forum, from http://it.coe.uga.edu/itforum/paper92/paper92.html.

Driscoll, M. (1994). Psychology of learning for instruction. Needham Heights, MA, Allyn & Bacon.

Driscoll, M. (2000). Psychology of learning for instruction. Needham Heights, MA, Allyn & Bacon.

Duchastel, P. (1998). "Prolegomena to a theory of instructional design."   Retrieved 4 October, 2009, from http://it.coe.uga.edu/itforum/paper27/paper27.html.

Ertmer, P. and T. Newby (1993). "Behaviorism, Cognitivism, Constructivism: Comparing critical features from an instructional design perspective." Performance Improvement Quarterly 6(4): 50-72.

Goldman, A. (1986). Epistemology and cognition. Cambridge, MA, harvard University Press.

Herrington, J., T. Reeves, et al. (2005). "Online Learning as Information Delivery: Digital Myopia." Journal of Interactive Learning Research 16(4): 353-367.

Mayer, R. (1996). "Learners as information processors: Legacies and limitations of educational psychology’s second metaphor." Educational Psychologist 31(3/4): 151-162.

Mishra, P. and M. Koehler (2006). "Technological pedagogical content knowledge: A framework for teacher knowledge." Teachers College Record 108(6): 1017-1054.

Pallas, A. (2001). "Preparing education doctoral students for epistemological diversity." Educational Researcher 30(5): 6-11.

Postman, N. (1995). The end of education. New York, Vintage Books.

Reigeluth, C. (1999). Instructional-design theories and models: A new paradigm of instructional theory. Mahwah, NJ, USA, Lawrence Erlbaum Associates.

Reiser, R. (2001). "A history of instructional design and technology: Part II: A history of instructional design." Educational Technology Research and Development 49(2): 57-67.

Sackney, L. and B. Mergel (2007). Contemporary learning theories, instructional design and leadership. Intelligent leadership: constructs for thinking education leaders. J. Burger, C. Webber and P. Klinck. New York, Springer: 67-98.

Seidel, R., K. Perencevich, et al. (2005). From principles of learning to strategies for instruction: empirically based ingredients to guide instructional development. New York, Springer.

Shulman, L. S. (1986). "Those who understand: Knowledge growth in teaching." Educational Researcher 15(2): 4-14.

Siemens, G. (2005). "Connectivism: A learning theory for the digital age." International Journal of Instructional Technology and Distance Learning 2(1).

Siemens, G. (2006, November 12, 2006). "Connectivism: Learning theory or pasttime for the self-amused."   Retrieved 9 September, 2009, from http://www.elearnspace.org/Articles/connectivism_self-amused.htm.

Siemens, G. (2006). Knowing Knowledge, Lulu.com.

Steffe, L. and J. Gale (1995). Constructivism in education. Mawah, NJ, Lawrence Erlbaum Associates.

Online learning better than blended learning?

There’s a bit of talk at my current institution about adopting “blended learning”. The Vice Chancellor had this to say and various other folk have been pushing the idea. However, I do hear that there has been some disagreement about the exact definition of blended learning being used by various parties.

I believe one definition being used is this one from Griffith University

Blended learning involves integration of different modes of delivery, models of teaching, and styles of learning through strategic and systematic use of technology, combined with the best features of face-to-face interaction.

A definition that retains a fairly central emphasis on face-to-face interaction.

This particular definition was always a bit problematic for me within the local context as a significant proportion of our students, and until recently our only growing market has been distance education students. i.e. students who are likely never to attend a campus.

The Vice Chancellor has been using the Wikipedia definition

Blended learning offers learners the opportunity “to be both together and apart. A community of learners can interact at anytime and anywhere because of the benefits that computer-mediated educational tools provide. Blended learning provides a ‘good’ mix of technologies and interactions, resulting in a socially supported, constructive, learning experience; this is especially significant given the profound affect that it could have on distance learning.

This definition seems to be more appropriate for the local context. It doesn’t treat face-to-face as anything special and also doesn’t use the “strategic” word which I somewhat detest.

As a colleague of mine pointed out

The term has been used in so many different ways it is almost useless. There should be a clear definition of CQU’s concept of blending learning so that we can all understand.

Online versus blended?

Interestingly, the folk pushing the former definition of e-learning have also been using the recent report from the US Department of Education that evaluates evidence-based practices in online learning as support for the view. This report has been making the rounds in the blogosphere and due to my focus on the PhD I’ve ignored it (the fact I’m currently commenting on this gives some indication of the quality of my focus on the PhD at the moment). However, I saw this summary this morning in my tweet stream and it has something to say on the question of blended learning and face-to-face.

I’ll include the quotes I found interesting, you can read the blog post and the report yourselves. And I certainly must find some time to read the report.

Blended learning versus online

Blended and purely online learning conditions implemented within a single study generally result in similar student learning outcomes.”

Control and reflection

“Online learning can be enhanced by giving learners control of their interactions with media and prompting learner reflection.”

Online versus f-t-f

one should note that online learning is much more conducive to the expansion of learning time than is face-to-face

Value of this sort of thing

Of course, this sort of meta-comparison is often not all that useful in improving practice. These evaluations are generally testing applications designed by academics motivated, for a variety of reasons, to engage in “good teaching”. The trouble is that because of the context they find themselves and a variety of other reasons most academics are not and, in many cases, cannot engage in “good teaching”.

For me the big problem is not really finding out what is “good teaching”. Though this is important. So whether online is better than blended is better than face-to-face, is not really all that important. Though it can be useful to be aware of these sorts of quotes to counter pragmatic quoting from proponents trying to boost a particular approach.

The big problem is how you get the vast majority of academics to improve their teaching, and keep improving it.

Herding cats and behaviour change

My focus for the last couple of weeks was a presentation – Herding cats and losing weight: How to improve learning and teaching. One description of the argument of that presentation is:

  • To improve learning and teaching within a university you have to change the conceptions (the behaviour) of the academics.
  • You don’t/can’t do that by “herding cats”, teleological, top-down approaches to change. i.e. most of what happens in universities at the moment.
  • Instead you have to do something much closer to effective approaches to losing weight. The basic point of losing weight is for people to change their behaviour so that they “eat less, move more”.

Focus on desirable futures, at the expense of the here and now

Today, via a tweet from David Gurteen which points to this blog post – “Behaviour change, revisited”. The message seems to resonate with the ideas from the presentation, particularly the following

Meanwhile, we’re so busy dreaming up desirable futures for each other, that we don’t notice all the subtle changes that are going on around us anyway. And while we craft our master strategies, we don’t even think about the little experiments we could make to nudge the system and see what happens.

The herding cats folk are so busy focused on the “desirable futures” that they forget the present, they aren’t connected with the present reality and don’t even know what small experiments might work. More on this in another post.

This idea seems to connect with a quote I used in the presentation from Mintzberg et al (2003)

To allow ourselves to be pulled by the concerns out there rather than being pushed by the concepts in here

Behavior change models

The first post above links to a broader conversation arising out of another blog post. Some interesting points in the post, and especially the comments, about change and change management.

Included in the comments is a link to this blog post talking about design thinking, social marketing and behaviour change. It includes a slideshare presentation titled Introduction to behavioural design

There are some connections here with what I’ve been thinking of. Oh, for more time.

References

Mintzberg, H., B. Ahlstrand, et al. (2003). Strategy Safari: A guided tour through the wilds of strategic management. New York, The Free Press.