The following is an early draft of the “lessons learned” section of chapter 5 of the thesis, the third last section that needs to be completed (to first draft stage). It still needs some work and completing the last two sections will probably lead to some changes, but it’s a start.
The basic aim of this section is to draw out reasons why the intervention (in this case the Webfuse system I designed) succeeded or failed. As I write this, pretty sure I haven’t finished.
Before attempting to describe the final ISDT arising from this work, this section seeks to reflect on the outcomes of the intervention in an attempt to understand how well the intervention achieved the changes sought and to understand the observed success and failures.
Relative unimportance of the technical product
From the perspective of data structures, algorithms, and bleeding edge technology Webfuse was not at all innovative. Use of scripting languages, relational databases, and open source applications to construct websites was fairly common and widespread. Nor would much of its implementation be considered theoretically correct by researchers focusing on relational databases, software engineering or computer science. For example, the schema used by Webfuse databases could not be described as being appropriately normalised. In addition, a common complaint about Webfuse has been that it was using technology that would not scale and that was not “enterprise-ready” (even though it could and did scale and support the enterprise). The questions of technical novelty, technical purity, or fulfilling arbitrary scalability guidelines had little or no effect on the success of Webfuse. The success of Webfuse arose from becoming, and being able to stay, an integral and useful part of the everyday life of the students and staff of the institution.
Webfuse was not a product
This emphasis on the characteristics of the technical product was also evident in the continual queries from colleagues asking when Webfuse would be sold or made available to other institutions. The ability and adoption of Webfuse as a product by other institutions was seen as a way for proving its success. This was based on the assumption that Webfuse, like all software, was a product that could be reused regardless of the organisational context. This was also the assumption that underpinned the development of Webfuse during the first phase of its development (1996 though 1999). One example of this is the observation that Webfuse was made available as an open source project for people to download in 1997.
A key characteristic of the second phase of Webfuse development was the recognition that the product and its features was not as important as how well its features matched the needs of the local context and continued to evolve in response to those needs. The most important part of Webfuse was the process, not the product. It was through this process of contextual adaptation that Webfuse became part of the way things were done at CQU, it became part of the culture. This tight connection with the institution meant that while the principles behind Webfuse and some of the applications might be useful at other institution, it was impossible to distribute Webfuse as software product. An understanding of this distinction improved the implementation of Webfuse. However, an inability to explain the importance of this distinction to various stakeholders contributed to the eventual demise of Webfuse and especially its ateleological process.
The importance of the pedagogue
Coates et al (2005) suggests that a recurrent message from educational technology research is that “it is not the provision of features but their uptake and use that really determines their educational value”. This message matches well with the experience of Webfuse. During the initial phase of Webfuse development described in Chapter 4, the provision of features in terms of various page types was not sufficient to generate use by academic staff and consequently any impact in terms of educational value. If the teaching staff responsible for a course did not use the provided features, or did not integrate them effectively into a course, there was no educational value. The pedagogue was of central importance in terms of any educational value arising from e-learning. It was through understanding and using this principle that Webfuse was able to become and everyday part of the practice of teaching academics.
Change takes time, familiarity, need, support, and adaptation
Many, if not all, teaching staff did not make decisions to adopt new educational practices and technologies immediately upon their release. Such adoption decisions occurred over varying periods of time as a result of a combination of individual contextual factors. Effective use of novel practices and technologies often lagged adoption by a number of years. The introduction of novel practices into an organisation generated a need for changes in organisational practices and support in order to become widely adopted and appropriately used. For example, the use of the course barometer feature was highest and most appropriate in 2002 and 2008 (see Figure 5.12) when use was encouraged and supported by organisational resources. In addition, as novel features become more widely used, there is a need to adapt those features to the requirements in response to lessons learned and changing requirements.
Helping people increases trust and knowledge
From 2000 onwards the Webfuse development staff also fulfilled the roles of system trainers and frontline helpdesk staff. Each of these roles are inherently challenging and attempting to balance the competing demands of each role adds further to the complexity. However, there were also a number of significant benefits that arose from this multi-skilling. These benefits included:
- Helpdesk staff with increased knowledge of the systems;
The helpdesk staff handling user problems had deep understandings of how the systems worked, what they could do, and how they could be manipulated. This deep level of knowledge enabled quicker and more flexible responses to problems.
- Increased ability for rapid changes; and
In some cases, those flexible responses involved quick modification of the Webfuse code to correct a problem or add a new feature. Such minor problems did not have to rise through a helpdesk escalation process before being remedied.
- Developers with increased knowledge of the needs and capabilities of the users.
The offering of helpdesk support and training sessions provided a deeper understanding of the capabilities and needs of both staff and student users that could drive the on-going design and development of Webfuse.
Each of these benefits combined to increase trust in the system and its direction, as evidenced by the increased use shown above and the following quote from a member of academic staff
my positive experience with other Infocom systems gives me confidence that OASIS would be no different. The systems team have a very good track record that inspires confidence
You can’t keep all the people happy
The experience with Webfuse from 2000 through 2009 has highlighted just how difficult answering the question – “Was Webfuse a success?” – actually is and how dependent it is upon the experiences and position of the person answering the question. In terms of success, it is possible to point to the statistics showing much higher levels of usage by staff and students. It is also possible to point qualitative comments from staff around trust and confidence and to formal management reports describing Webfuse as “[t]he best thing about teaching and learning in this faculty in 2003”. At the same time, it’s possible to point to consistent arguments from central IT staff that Webfuse was a shadow system that duplicated existing systems and was subsequently inefficient and wasteful (Jones, Behrens et al. 2004). There were also comments from one senior member of staff in 2004 suggesting that Webfuse had made no significant difference to learning and teaching at CQU.
Ateleological processes don’t fit in a teleological environment
Webfuse experienced its greatest levels of support and improvement during the period from 2000 through 2004. During these years the Faculty of Informatics and Communication (Infocom) – which supported Webfuse – was undergoing significant growth in student numbers, complexity, and available resources. At the same time, Infocom had a Dean who had publicly expressed support (Marshall 2001) for a more ateleological approach to organisational and systems development, and was comfortable with that approach within Infocom.
From 2004 onwards there were a number of changes within CQU, including: (1) changes in faculty and institutional leadership; (2) changes in student enrolment profile raising concerns about faculty and institutional funding; and, (3) an organisational restructure resulting in increased centralisation and/or out-sourcing of services. These changes led the institution toward a much more teleological approach to systems development and support. Under these conditions the ateleological Webfuse process was seen as wasteful of resources and, to some extent, nonsensical.
Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.
Jones, D., S. Behrens, et al. (2004). The rise and fall of a shadow system: Lessons for enterprise system implementation. Managing New Wave Information Systems: Enterprise, Government and Society, Proceedings of the 15th Australasian Conference on Information Systems, Hobart, Tasmania.
Marshall, S. (2001). Faculty level strategies in response to globalisation. 12th Annual International Conference of the Australian Association for Institutional Research. Rockhampton, QLD, Australia.