CQUniversity is in the process of evaluating two open source learning management systems for adoption by 2010. We’re at the stage of finalising the criteria to be used to evaluate the two systems. The purpose of this post is to outline some thoughts about how to evaluate the evaluation criteria.
What should such criteria cover? What are the common problems? That sort of stuff. This is an attempt to think about this before I look in detail at and provide feedback on the proposed evaluation criteria we’ll be using. Due to time constraints it isn’t as informed by literature as it should be. It’s a simple discussion of the things I can come up with late on a Sunday night, supplemented with a quick skim of the results of a couple of Google searches.
Deeper philosophical problems
I should start by saying that at a fundamental level I don’t believe the upfront evaluation of a large integrated system, like a learning management system (be it open source or commercial), is the most effective way to support learning and teaching through the application of information technology.
I’ve also read enough of the information systems discipline’s research literature to know that the quality of the decision making that occurs during these sorts of “rationale” evaluation processes is anything but rational. There is a long list of research projects, including some that examined decisions at Australian universities, that suggest that such processes are far from rational.
However, the organisation has decided to choose between one of two systems and in my role I need to contribute as best I can (because my contributions will be as less than rational as everyone elses) to the positive outcome of the process.
However, it’s probably worthwhile to outline some of the problems this type of evaluation faces, if only to be aware of them. They include
- How can a small number of folk with limited backgrounds make estimates/guesses about the future organisational needs?
- How can the intangible costs and benefits of the change and the systems be understood by these folk?
- Some aspects of e-learning remain innovative, by definition this is different and will have unknown impacts.
- How to meaningfully quantify and compare the benefits of a particular system against another, especially given the very different perspectives of those involved. For example, the IT person will look for certain types of benefits, the central L&T support person another and the many different academics using the systems another set.
- Which doesn’t even mention the most important group of users of these systems, the students, and how you involve, understand and evaluate for the benefits that they are interested in.
- How do you identify and get consensus around the definition of what an effective learning management system is?
- When examining costs, how can you properly evaluate the problem of “hidden costs”. For example, background IT management costs and costs associated with end-users helping each solve system problems.
- Elsewhere I’ve argued that L&T at a university is an example of a complex system. i.e. one in which cause and effect cannot be properly predicted. You can’t predict what might happen as a result of the implementation of a new system.
How to evaluate
Cronholm and Goldkuhl (2003) identify 3 types of evaluation “processes”
- Goal-based evaluation
The goals of the organisation drive the evaluation.
- Goal-free evaluation
No such explicit organistional goals exist and evaluation usually proceeds in an “inductive and situationally driven strategy”. i.e. no pre-set goals are identified, instead as the evaluation progresses, usually involving a broad number and spread of participants, knowledge of the object of study emerges during the evaluation.
- Criteria-based evaluation
Some collection of general criteria (not specific to the organisational context) are used to compare systems. The assumption here is usually to see if the IT system supports the activities performed by the business (assuming you know what it is)
What to evaluate
Cronholm and Goldkuhl (2003) identify 2 ways to identify what to evaluate
- IT-system as such.
Essentially, without the users. Don’t actually get in and use the system. Simply use one of the processes from above.
- IT-system in use.
Study the system in a use situation with users interacting with it.
They then go onto combine the identified approaches to identify and describe six different types of evaluation process. The examine each of these types against the following characteristics
- Main perspective
- What to achieve knowledge about
- data sources
- Deductive or inductive
- Who will participate
- When to chose this type
Potential thoughts for CQUni’s evaluation of LMSes
Based on the above and a little bit of reflection the following is a list of random thoughts that might need to be considered when looking at CQUni’s evaluation criteria/process for its next LMS.
- Does the evaluation criteria connect with CQUni’s strategic goals? i.e. is it goal-based or simple criteria-based (my understanding of the process is that it is not a goal free process)
- Which of CQUni’s strategic goals are impacted upon or potentially enabled by this choice?
- How do the selection criteria connect with established institutional understandings of “good” L&T – e.g. Chickering and Gamson’s 7 principles.
- The “what” is being evaluated is a simplified “IT-system in use” hosted by an external provider with only a small number of staff and no students. What are the limitations implicit in such an approach and how to counter them?
- Do the technical criteria talk about the system in use and not an “objective” version of the system? i.e. you can’t say that System X is scalable, secure etc. It’s the specific details of how System X is implemented within a particular organisation that is scalable, secure etc.
- Is it clear who will be evaluating what and based on what evidence? i.e. can folk simply choose a figure/result or do they have to provide evidence?
- Is it clear how all the different perspectives will be gathered, sifted and used in the decision making process?
- Get a better handle on what people are doing now in order to identify a better understanding of which system fits current practice.
- Ensure that what students see as important, is in someway included in the evaluation.
Not all of these will make sense tomorrow.
Stefan Cronholm, Goran Goldkuhl (2003), Strategies for Information Systems Evaluation- Six Generic
Types, Electronic Journal of Information Systems Evaluation, 6(2), pp 65-74