Mapping the digital practices of teacher educators: Implications for teacher education in changing digital landscapes

Paper presented at SITE’2016. One of three papers awarded the Ann Thompson TPACK Paper Award.

Authors: Peter Albion, Amanda Heffernan, David Jones

Abstract

Almost 40 years since the first personal computers appeared in classrooms Education is still awaiting transformation on the scale experienced in other parts of society. The replacement of digital immigrant teachers by a younger generation of digital natives has not made the anticipated difference. That metaphor is discredited and new perspectives are needed. In this paper the metaphor of digital visitors and residents is adapted to support examination and mapping of the digital practices of teacher educators according to the traces they leave in the digital landscape and levels of modification to tools as supplied. Questions are asked about the degree to which teachers and teacher educators need to modify tools or create their own in order to better adapt ICT in support of learning and teaching.

Priming the Future

In his introduction to an Australian Council of Learned Academies (ACOLA) report on Technology and Australia’s Future (Williamson, Raghnaill, Douglas, & Sanchez, 2015), Australia’s Chief Scientist commented that humanity would be better served by ‘future-priming’ rather than ‘future-proofing’ (Chubb, 2015) since the future is something we create through our actions in the present rather than necessarily a threat. The report itself laid out four key messages: technology will drive long-term economic growth, technology will transform the workforce, Australia can leverage technology change for societal benefit, and forecasting future technology development is challenging. In essence, we are in a period of rapid technological change and future citizens need to be prepared through their education to adapt creatively and manage, rather than be shaped by, technological change.

If our future generations are to be prepared for adaptability to technological change as suggested, then education in the present must deal with a cascading series of challenges. Changing the capabilities of graduates from our schools and universities requires that the teachers responsible for their education must change their practice. For teachers to change, their own preparation and professional development must change and that requires change in the practice of teacher education and teacher educators. Hence questions about the technological capabilities required of teachers inevitably raise questions about the capabilities required of teacher educators.

As recognized in the ACOLA report (Williamson et al., 2015), many of the technological changes to which we must adapt are based on digital technologies in whole or part. Hence there is increasing emphasis on digital technologies in education. The Australian Curriculum (ACARA, 2015) includes digital technologies in multiple guises. Information and Communication Technology (ICT) is included as one of 7 general capabilities, incorporating investigating, creating and communicating with ICT, managing and operating ICT and applying social and ethical protocols and practices. The curriculum lays out a learning continuum and expectations for how ICT should be represented within the 8 learning areas. One of those areas is Technologies for which the recently endorsed curriculum includes two related subjects to be taught from Foundation to Year 10. Design and Technologies is an updated version of a subject that has been taught around the country in varying forms over the past couple of decades. Digital Technologies is a new departure with a focus on computational thinking in conjunction with design and systems thinking. It includes some required study of programming alongside study of hardware, including robotics, and data design. Over and above the inclusion of ICT in the curriculum there is a general expectation that ICT will be used effectively for learning and teaching across all curriculum areas. One indicator of the prevalence of that idea was the Digital Education Revolution, a national project initiated in 2008 with a series of measures including funding to provide laptop computers to all students in Years 9 to 12 (Jamieson-Proctor et al., 2014).

Priming Teachers for a Changing Future

There are teachers in Australian primary schools doing excellent work with ICT in support of learning and teaching across the curriculum while at the same time embedding the ICT general capability and engaging with elements of the new Digital Technologies curriculum. However, it seems certain such teachers are a small minority and that many primary school teachers risk being overwhelmed by the perfect storm of ICT general capability, Digital Technologies and ICT-enhanced pedagogy. A requirement to engage seriously with coding, which will be literally a foreign language for them, may push them too far. Schooling has often been a convenient location to send problems identified in the wider society. The consequence has been an increasingly crowded curriculum and uncertainty about the knowledge required of teachers.

The challenges of preparing pre-service teachers (PSTs) to enhance learning and teaching with ICT have been widely canvassed and the associated challenges of supporting teacher educators to model the behaviors desired from PSTs have been examined (Jones, Heffernan, & Albion, 2015). Those challenges have been described as “wicked problems” (Mishra & Koehler, 2007) for which solutions are elusive. Nevertheless, “Integrating Technology in Teacher Education” has been identified as a “solvable challenge”, one that “we understand and know how to solve” (Johnson et al., 2015), with the proposed solutions ranging across competency training based on national standards, using ICT to recruit and train teachers for technology integration, and specially produced training resources. Variations of these approaches have been tried over the past 30 years but the challenge persists and it is tempting to ask how the new projects are different.

Early observers of the limited application of ICT by teachers in their classrooms easily leapt to the conclusion that the change was difficult for teachers unfamiliar with ICT but that the arrival of a new generation of teachers who had grown up with ICT would make the difference. For several years this idea drew support from the meme about digital natives and immigrants (Prensky, 2001) but subsequent examination of the evidence debunked that concept, likening it to a form of ‘moral panic’ (Bennett, Maton, & Kervin, 2008). Other researchers have confirmed that the age-related fault lines suggested by Prensky’s metaphor do not exist. In reality, the rate of development in ICT means that we are all perpetual immigrants required to adapt to a constantly changing and unfamiliar landscape.

Granted that adaptability is a desirable quality in a time marked by rapid technological change (Williamson et al., 2015), there are questions around the nature and degree of adaptability required of graduates from schools, their teachers, and the teacher educators who prepare them. Tools and practices that are familiar as analogues can be adopted easily. For example, word processing was widely adopted based on its similarity to using a typewriter and email and other communication tools have followed, though there is often little sophistication of use and advanced features are mostly neglected. Spreadsheets and databases have been much less widely adopted and very few ‘immigrants’ have attempted to learn a local language and engage in coding. What level of adaptation will, or should, be required of teachers and teacher educators? Is it realistic, as seems to be suggested by the Queensland Government (DET, 2015), to expect all to engage in learning and teaching coding? If coding is the new literacy required by every student, then how will teachers and teacher educators make the necessary transition? The type of ‘moral panic’ induced by the native/immigrant meme can limit our ability to understand what motivates individuals’ engagement with digital technologies and make it more difficult to understand and develop the capability and desire for individuals to engage with digital technologies (Connaway, White, Lanclos, & Le Cornu, 2011).

White & Cornu (2011) introduced the Digital Visitor and Resident (V&R) model as a framework to understand how people engage with digital technologies, especially the participatory web. The V&R model does not assume that either age or gender determines engagement with digital technologies. Instead it focuses on examining what is done with digital technologies (digital activities) and why. The V&R model uses a space/place metaphor. A Digital Visitor sees the digital space as a collection of disparate tools that are used to achieve specific tasks before beating a hasty retreat, leaving little evidence of having entered the digital space. A Digital Resident sees the digital space as an environment to inhabit, build relationships with other people, and project identity(ies). For a Digital Resident there is value in inhabiting the digital space. The visitor and resident modes are not exclusionary, with individuals likely to practice a mixture of both, dependent on the goals or tasks they have set themselves from time to time. The V&R model was used as the primary framework for a three-year longitudinal study (Connaway et al., 2011) to understand the motivations behind, and types of engagement with, digital technologies by a sample of students and scholars from late secondary students through experienced academics. This study led to the development of a V&R mapping tool that has been used to explore how students and scholars are engaging with the digital services provided by institutions.

This paper adopts and adapts the V&R mapping tool to explore the motivations and engagement with a digitally rich learning space by teacher educators with the hope that such an approach can help understand how and why teacher educators are engaging with digital technologies. Given the growing level of interest in computational thinking and the identified importance of the protean nature of digital technologies (Jones et al., 2015) this project has a particular interest in exploring how and why teacher educators engage in practices where they modify the digital tools provide by institutions, and adopt or create new digital tools to serve their pedagogical purposes.

The remainder of the paper offers a brief description of the participants and process used for data collection followed by a description of the mapping dimensions as adapted for this study. It will then present some examples of stories and describe their mapping on the axes before proceeding to derive some implications for teacher education.

The Queensland Government has recently launched a new action plan for its next phase of educational advancement (DET, 2015). One aspect of that plan includes a commitment to implement the new Digital Technologies curriculum from 2016 with a focus on the “coding and robotics skills needed by students for their future”. According to the website, “coding has quickly become the new literacy and a ‘must have’ for every student”. The launch included a hashtag, #codingcounts, and invitations to “join the conversation” by completing online surveys.

Understanding and mapping digital practices

Participants and process

The data for this research are provided by stories of the digital practices undertaken by each of the authors within a teacher education program that has up to 70% of its students studying some subjects online (Albion, 2014) and has identified digital learning in various forms as a strategic priority. This particular sample is seen as likely to provide useful insights due to the apparent digital richness of the learning environment and the diversity of the three authors. Given the strategic importance of online learning in this context, teacher educators are required to make significant use of digital technologies. However, as reported previously (Jones et al., 2015) there has been the need to engage in a range of practices to address limitations in institutional practices and technologies. Each of the authors also represents a cross section of experience (20+ years in teacher education; 4 years in teacher education; 3 years in teacher education) and digital literacy (proficient user but no formal qualifications; graduate diploma in Information Technology; bachelor degree in computer science and PhD in information systems).

Each author was asked to generate a list of stories where they have modified (broadly defined) digital technologies while undertaking their role as teacher educator. Each story used a consistent format of four parts: a descriptive title; a description of the change made; an explanation of the rationale; and, a summary of the outcomes. All stories were added to the same Google document (http://bit.ly/1nuLZdH) allowing each participant to read the others’ stories. Each participant then located his or her stories on a map (Figure 1) adapted from the V&R mapping process (White, Connaway, Lanclos, Hood, & Vass, 2014). As an exploratory process the story creation and mapping process was iterative. The final maps (Figures 2-5) from each author were then used as the basis for discussion and analysis.

The map

While informed by the original V&R mapping process (White et al., 2014) and attempting to retain its overall goals to understand digital practices, the change in focus in this work has led to adaptation of the V&R map (Figure 1). The V&R map remains as a Cartesian graph with an X- and Y-axis. The X-axis retains the Visitor and Resident scale, but with the scale understood to start with individual use of a tool and then proceed through increasingly large groups of people including: individuals; small groups; whole course cohorts; multiple course cohorts; and eventually onto the open web. Given our focus on why and how teachers are modifying digital technologies, the original scale of personal/enterprise for the Y-axis is replaced by a scale indicating the level of modification summarized in Table 1. While the focus in this project is on how digital technologies are modified, “use” is retained in the scale so that the map can be part of broader explorations of the digital practices of teachers. Table 1 describes the levels of modification used on the Y-axis of the map and Figure 1 represents the template used during initial mapping of stories. The mapping process was iterative, with the template and dimensions subject to refinement as stories were mapped.

Table 1: Levels of modification for Y-axis

Item Description Example
Use Tool used with no change Add an element to a Moodle site
Internal configuration Change operation of a tool using the configuration options of the tool Change the appearance of the Moodle site with course settings
External configuration Change operation of a tool using means external to the tool Inject CSS of Javascript into a Moodle site to change its operation
Customization Change the tool by modifying its code Modify the Moodle source code, or install a new plugin
Supplement Use another tool(s) to offer functionality not provided by existing tools Implement course level social bookmarking by requiring use of Diigo
Replacement Use another tool to replace/enhance functionality provided by existing tools Require students to use external blog engines, rather than the Moodle blog engine.

 

V&R modification map

Figure 1: Axes used for mapping stories

Stories and themes

Part of the ‘wicked problem’ of ICT in education (Mishra & Koehler, 2007) is the particularity of each context. Hence it is important that teachers be able to modify or contextualize their learning environment. The mapping exercise encouraged the authors to review the various ways each of us has modified our course learning environments, and identify that while some of our practices skewed toward the ‘resident’ dimension, our behaviors generally remained primarily in the ‘visitor’ dimension. A wider spread in behavior was identified in the modification dimension, with practices evident from the lower end of the scale and the ‘use’ of tools as they were initially intended, moving through to the higher end of the scale and the ‘replacement’ of tools that were deemed to not meet our needs.

David's V&R modification map

Figure 2: David’s V&R modification map

Peter's V&R modification map

Figure 2: Peter’s V&R modification map

Amanda's V&R modification map

Figure 3: Amanda’s V&R modification map

Minor Modifications: Internal and External Configuration of Tools

 

The stories provided us with examples of practice at each level on the modification scale. For example, at the lower end of the scale, Amanda achieved minor renovation of the learning space through the internal configuration of the layout of the weekly study schedule page on her course site within the Moodle Learning Management System (LMS). This is traditionally an area where students are provided with an overview of the semester’s work and default headings provide students with information about required readings, module work, and assessment information. Amanda adjusted the traditional headings and layout to provide two streams of sequencing throughout the course, directing students towards course content as one stream and employment information as the other. As a result, she was able to configure the tool to better meet the needs of her course and its participants, all of whom are in their final semester and looking towards employment and career development as well as their coursework.

 

At the next level, David and Peter both provided examples of practice where they used external configuration of tools to meet their needs. David identified the use of jQuery and CSS to modify the operation of the course environment’s university-wide standardized look and feel. The key outcome of this configuration was that ongoing cultivation of the learning environment was not halted by the university’s new streamlined look, and students were able to access the course content in a way that was more functional and efficient. Similarly, Peter made use of AppleScript/JavaScript to more efficiently organize student groupings within the online environment based on data managed in a spreadsheet. The external configuration of the grouping option enabled Peter to have greater control over group composition according to factors that were identified as being important (such as geographic location, age, or other demographic factors).

 

These minor modifications at the internal and external configuration levels are representative of some of the stories shared by the authors, and the practices above are indicative of renovations being made for pedagogical as well as administrative purposes, enabling us to configure the learning environment for our needs. More advanced modifications were also identified within the stories, resulting in higher levels of customization and the tailoring of learning environments to our needs, as well as the needs of our students.

Major Modifications: Customization, Supplement, Replacement of Tools

During the mapping and analysis of practices, it became evident that David and Peter worked more in these areas of customizing, supplementing, and replacing tools than Amanda did. Their level of expertise and experience no doubt plays some part here in enabling higher levels of customization of tools through the use of coding. However, all three of the authors was able to identify some practices at the replacement level, wherein an internal tool was replaced with an external tool that better suited our purposes.

David and Peter identified practices of customization, providing examples of modifying codes within the tools to meet their needs. Peter described his use of HTML and CSS to arrange the display of Twitter and Diigo in boxes on his Moodle sites. In previous semesters this had been a simple inclusion but the shift to the new look and feel of the institution’s LMS removed the ability to use pre-existing blocks to do this. Instead, Peter had to modify the code of the page, using the browser inspection tools to deconstruct the HTML and CSS in order to customize the layout of the page and place the code for the required content.

Similarly, David’s practices at the customization level included the installation of his BIM activity module, enabling students to use blogs outside of the institution’s own tools and register them within his course environment. David outlined a number of reasons for the creation and implementation of this activity module, ranging from pedagogical reasons (enabling his students to connect with networks outside of the course) to administrative processes (the inclusion of marking interfaces and options into the module). Even though David is the designer and maintainer of the BIM activity module, the mismatch between the functionality of this tool and the requirements of the learning design led David to undertake some supplement level modifications.

At this higher level of modification, David and Peter detailed practices within the supplement level, many of which minimized administrivia by enabling system tools to interface more efficiently. For example, David used his coding skills to develop a collection of scripts and tools referred to as ‘know thy student’, providing him with information about students with the ease of a single click, which would otherwise take over ten minutes and multiple webpages to unearth (Jones and Clark, 2014). This supports David’s teaching in both a pedagogical sense (enabling him to better know his students and meet their needs more effectively), and in a practical sense, ensuring the value of his time is maximized by having this information readily and easily available when needed. David’s use of practices at the supplement level frequently resulted in streamlined processes that saved time and unnecessary additional work, with another practice being identified that more easily supported the process for finalizing course results. Through a collection of Perl scripts and spreadsheets, David is more easily able to undertake the potentially onerous process of finalizing results in a large course with over 300 students enrolled. David also had a number of other supplement stories with names including: ‘Diigo’; ‘Google docs’; ‘Gradebook fix’; parts of the ‘Book authoring process’; and, ‘A duplicate Moodle’.

Peter’s practices at the supplement level were equally effective in enabling efficient use of time to undertake administrative processes. He was able to develop a Greasemonkey script to assemble and display simple statistics for comparing results from different markers in his courses. With 150 students over 4 offers, and markers being typically casual staff with limited experience of the course, Peter felt it necessary to guard against any systematic advantage that might occur for students in one or other offer because of marker differences. Rather than the traditional method for comparing marks by markers, which requires exporting data from Moodle to a spreadsheet and constructing formulas to generate statistics, Peter developed a script to perform these calculations. Again, this is pedagogically sound and ensures equity in marking and assessment for all students, regardless of their circumstances. The ability to renovate at this level saved Peter a significant amount of time.

Examples of practices at the final level of the modification scale, replacement involve the teacher replacing an internal tool with a similar external tool. Examples of these practices were very limited in our story collection. One example of replacement was of Amanda making use of Vimeo rather than the institution’s internal media repository to organize and share videos so that other users, including students’ mentor teachers and past students, can access certain videos and presentations. The use of Vimeo also allows Amanda to embed videos within pages and reflects the internal configuration level, where existing tools are enhanced. David had two – somewhat similar – stories of replacement both based on the use of external blog engines to replace institutional systems. In ‘Escape the LMS` the primary course site for a Master’s course titled Networked and Global Learning was moved from the institution’s Moodle to a blog hosted on WordPress.com. In ‘Student space as their space’ students in an undergraduate ICT and Pedagogy course were required to create and use a blog on an external blog engine for reflection and building a personal learning network, rather than use an institutional e-portfolio or the Moodle blog. It is arguable whether these two stories are truly examples of replacement, rather than stories of how to supplement.

Discussion and implications

The aim here has been to use the metaphor of digital visitors and residents to examine and map the digital practices of teacher educators, in particular, those practices that involve the modification of provided tools or the creation of new tools. This is important due to a range of factors, including: the perceived value of teachers being able to modify or contextualize their learning environment; the protean nature of digital technologies; the growing prevalence of digital technologies within curriculum, learning and teaching; and, the argument that being successful in the future will require such skills. The focus here has been on the stories of digital modification practices by three teacher educators operating in a learning environment where use of digital technologies is compulsory. These stories have been examined and mapped using a modified Digital Visitor and Resident mapping process.

The analysis has revealed that all three teacher educators – regardless of perceived digital literacy – have engaged in a range of practices where the digital environment was modified. The greater digital knowledge of two of the teacher educators did increase the breadth and complexity of modifications. While there a few examples of modification at the replacement level, most modification was at the supplement level or below. By default, the digital learning spaces offered by the institution are limited to course participants; hence they are not part of the open web. This is evident in that most of the modification stories tended to congregate toward the Visitor end of the spectrum. It is interesting to note that all of the stories of replacement involved moving learning out onto the open web. Looking more closely at the stories suggests that there were two underpinning reasons for the modification activities. The first was to improve the efficiency of institutional systems or practices, and the second was to enhance learning through specific learning activities not directly supported by institutional systems.

This suggests that there is value in teachers being able to engage in digital modification practices to customize and contextualize the digital learning environment to the needs of themselves and their learners. It raises questions about how broadly digital modification practices are amongst teachers, the outcomes of those practices, and how any perceived inability to engage in digital modification practices is impacting student learning and the teacher experience. Experience engaging in these digital modification practices suggests that institutional systems and policies are not always able to recognize the need for teachers to engage in digital modification practices, let alone allow and enable such practices. What is the impact of these limitations on digital modification? Do digital modification practices become more important only in a learning environment – like that described here – where the use of digital technologies is a compulsory part of the learning and teaching experience? What, if any, digital modification practices do teachers in a more blended environment engage in?

We are operating in a learning space where we must use digital technologies. We have no choice. This is different from many other learning spaces. There is a chance that increasingly more learning spaces will have some level of compulsory digital technologies. The stories and maps arising from this work indicate that it will be likely that teachers will have to engage in activities that modify the provided digital technologies.

This work is exploratory. As such any findings may be specific to the particular institutional context and the three teacher educators. The broader value of these findings will need further testing and consideration but the V&R lens seems useful in terms of revealing new insights into educators’ use of ICT.

References

ACARA. (2015). The Australian Curriculum.  Canberra: Commonwealth of Australia (Australian Curriculum, Assessment and Reporting Authority) Retrieved from http://www.australiancurriculum.edu.au.

Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology, 39(5), 775-786. doi: 10.1111/j.1467-8535.2007.00793.x

Connaway, L. S., White, D., & Lanclos, D. (2011). Visitors and residents: What motivates engagement with the digital information environment? Proceedings of the American Society for Information Science and Technology, 48(1), 1-7. doi: 10.1002/meet.2011.14504801129

Chubb, I. (2015). ‘Technology and Australia’s Future’ report launch, from http://www.chiefscientist.gov.au/2015/09/speech-technology-and-australias-future-report-launch/

DET. (2015). Advancing education: An action plan for education in Queensland.  Brisbane: The State of Queensland (Department of Education and Training) Retrieved from http://advancingeducation.qld.gov.au/.

Jamieson-Proctor, R., Redmond, P., Zagami, J., Albion, P., & Twining, P. (2014). Redefining education for the digital age: a snapshot of the state of play in three Queensland schools. Paper presented at the Australian Computers in Education Conference 2014, Adelaide, SA.

Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2015). The NMC Horizon Report: 2015 K-12 Edition. Austin, Texas: The New Media Consortium.

Jones, D., Heffernan, A., & Albion, P. (2015). TPACK as shared practice: Toward a research agenda. In L. Liu & D. C. Gibson (Eds.), Research Highlights in Technology and Teacher Education 2015 (pp. 13-20). Waynesville, NC, United States: Association for the Advancement of Computing in Education (AACE).

Jones, D., & Clark, D. (2014). Breaking BAD to bridge the reality/rhetoric chasm. In B. Hegarty, J. McDonald, & S. Loke (Eds.), Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite Dunedin 2014 (pp. 262–272). Dunedin. Retrieved from http://ascilite2014.otago.ac.nz/files/fullpapers/221-Jones.pdf

Mishra, P., & Koehler, M. J. (2007). Technological Pedagogical Content Knowledge (TPCK): Confronting the Wicked Problems of Teaching with Technology. Paper presented at the Society for Information Technology and Teacher Education International Conference 2007, San Antonio, Texas, USA.

Prensky, M. (2001). Digital natives, digital immigrants Part 1. On the Horizon, 9(5), 1, 3-6.

Selwyn, N., & Bulfin, S. (2015). Exploring school regulation of students’ technology use – rules that are made to be broken? Educational Review, 1911(October), 1–17. doi:10.1080/00131911.2015.1090401

White, D. S., & Le Cornu, A. (2011). Visitors and Residents: A new typology for online engagement. First Monday, 16(9). doi: 10.5210/fm.v16i9.3171

White, D., Connaway, L., Lanclos, D., Hood, E., & Vass, C. (2014). Evaluating digital services: a Visitors and Residents approach. Retrieved October 22, 2015, from http://www.jiscinfonet.ac.uk/whole-infokit/?infokit=10856

Williamson, R. C., Raghnaill, M. N., Douglas, K., & Sanchez, D. (2015). Technology and Australia’s Future: New technologies and their role in Australia’s security, cultural, democratic, social and economic systems. Melbourne: Australian Council of Learned Academies.

Wright, F., White, D., Hirst, T., & Cann, A. (2014). Visitors and Residents: mapping student attitudes to academic use of social networks. Learning, Media and Technology, 39(1), 126–141. doi:10.1080/17439884.2013.777077

 

Predicting System Success using the Technology Acceptance Model: A Case Study

Behrens, S., Jamieson, K., Jones, D., & Cranston, M. (2005). Predicting system success using the Technology Acceptance Model: A case study. In 16th Australasian Conference on Information Systems (Paper 70). Sydney. Retrieved from http://aisel.aisnet.org/acis2005/70/

Abstract

Determining what makes an Information System (IS) successful is an ongoing concern for both researchers and practitioners alike. Arriving at an answer to this problem is compounded by the subjective nature of success and therefore trying to make judgements of what is and is not a success is problematic. Despite these difficulties system use has become more accepted as a measure of system success. Following this logic if a system is accepted it will have a higher likelihood of being used and therefore impact positively on success. The Technology Acceptance Model (TAM) is one of the more widely accepted theoretical frameworks that has been used to measure system acceptance. This paper combines the TAM, as the theoretical framework, with case study research to provide a more holistic account of why a specific IS, an online assignment submission system, has become successful. Initial findings suggest that the TAM measures of perceived usefulness and perceived ease of use are effective predictors of systems success.

Introduction

Measuring success within IS has been a concern for those within the discipline since its inception. Although success is complex and therefore difficult to measure researchers have made efforts in doing so. Traditionally these measurements focus on delivering a functional IS product within certain economic and temporal constraints. Despite this bias there is evidence to suggest that a more accurate measure of success may lie within the realms of system use. Based on the logic that a system must first be accepted to be used ensuring acceptance should increase the probability of system success. One of the more popular theoretical frameworks that predicts system acceptance of technology is the Technology Acceptance Model (TAM). We use this model to try and investigate why a specific IS innovation in use at Central Queensland University (CQU) has become so popular.

Davis et al.’s (1989) work on the TAM Information Systems theory, is a user centred approach which has gained popularity as a measure of technology acceptance. TAM suggests that when users encounter a new IS innovation there are two main factors which will influence how and when they will use it. These are perceived usefulness and perceived ease-of-use. Perceived usefulness is “the degree to which a person believes that using a particular system would enhance his or her job performance” (Davis 1989). Perceived ease-of-use is “the degree to which a person believes that using a particular system would be free from as the Unified Theory of Acceptance and Use of Technology (UTAUT) (Venkatesh and Davis 2000) there currently exists only one study confirming its validity and robustness. TAM on the other hand has been tested by many more researchers (Adams et al. 1992, Hendrickson et al. 1989, Segars and Grover 1993, Subramanian 1994, Szajna 1994) with different populations of users and IS innovations.

Due to the testing and support of the model by others we rely on the original TAM proposed by Davis (1989) rather than the extended model of the UTAUT (also known as TAM2) to measure technology acceptance. In this study we apply the model to a twelve year old IS, OASIS (Online Assignment Submission, Infocom System) in use at CQU. We apply the TAM measures of perceived usefulness and perceived ease of use to two different groups; staff non-users and staff users. The non-user group is examined due to their potential to become users and therefore impact on the continued growth of the system’s popularity. The user group is examined due to their importance in maintaining the current level of system use. Preliminary analysis into these two groups through the use of a case study approach reveals that the TAM measures appear to be useful predictors of a successful system. Our study also suggests that the evolutionary development model adopted by the system support team may have impacted positively on user perceptions and beliefs of system usefulness and ease of use.

Research approach

Rationale for using a case study approach

This research focused on determining the significant system acceptance factors that have contributed to the success of OASIS. This level of detail allows us to provide a more complete description and explanation of this particular innovation’s success. We believe that such a holistic focus of success using a proven IS is beneficial for researchers and practitioners alike. Although TAM has been the subject of investigation for much research, many of these studies are limited in several respects. These deficiencies include issues such as the strictly quantitative nature of the research with a focus on the adoption of simplistic technologies such as voicemail, email (Adams et al. 1992, Davis 1989, Segars and Grover 1993), word processing, spreadsheet and graphics software (Adams et al. 1992, Bagozzi et al. 1992). Our study helps to address these limitations in the literature by providing an in-depth qualitative study of a more complex non-mandatory system. For practitioners it provides a unique insight into an authentic and successful IS implementation. Given our main requirement of an in-depth investigation we used the case study approach. This decision is in accord with recommendations from proponents of the case study approach, for example Hamel (1993), Yin (1994), and Stake (1995).

Site selection – unit of analysis

Selection of the research site is the most critical decision in the analytic process of case study research (Hamel 1993). For our investigation, the choice of CQU as our site was a relatively simple matter. This was because CQU met all three of the main selection criteria; ‘no real choice’, suitability and pragmatism (Denscombe 1998). Firstly the site represented a unique opportunity for study. All researchers work at CQU and were able to observe the successful adoption of OASIS, a ‘home grown’ IS used in the support of teaching and learning, as well as various other information system adoption failures. The success of OASIS at CQU was an event that could not be ‘planned or created’ (Denscombe 1998). It therefore represented a ‘one-off chance’ for us as researchers to gain insight into why OASIS has become so successful where other ISs had failed. Consequently there was no real element of choice in ‘deciding’ on CQU as an appropriate site for study. Secondly the site was suitable due to the relevance of the case for testing previous theory. TAM is the theory we believe to be relevant in predicting the success of this system due to its ability to measure user acceptance. Through our own observations of OASIS and a review of the literature we believe that the success of OASIS at CQU contains ‘crucial elements’ (Denscombe 1998) of being a successful IS innovation which can test the theory of TAM. Finally the case is both intrinsically interesting and convenient for investigation. Although these final pragmatic considerations are not enough to choose the site on their own they added to the research experience allowing for a more in-depth study, hopefully appealing to a wider audience (Denscombe 1998).

Case study design type

Our case study is an investigation into why a specific information system, OASIS, is a success. Due to the formation and identification of TAM as the theoretical framework that guided our research, we utilised a descriptive case study design (Berg 2004). Our design follows the advice given by Yin (1994) and includes the five components he deemed necessary; study questions, theoretical framework, identification of the unit of analysis, logical linking of the data to the theory and the criteria for interpreting the findings. The first three (already previously stated) indicated the data we needed to collect and the last two how we would use the data. Although, according to Yin (1994) the last two are the least well developed in case studies we lay the foundations for this analysis early in our research design. Linking the data to the theory was to be done through a “pattern matching” technique (see Donald Campbell 1975 via Yin 1994). This is where several pieces of information from the same case may be related to some theoretical proposition. In our research we would determine whether the data matched or did not match the propositions put forth by the TAM theory through visual examination. The last component of determining the criteria for interpreting the findings is the most difficult of all (Yin 1994). Unfortunately there are very few guidelines on how to arrive at such a
criteria and all that exists is Yin’s very brief advice on matching rival theories. Given these difficulties we decided that we would rely on an “all or nothing” approach similar to Yin’s advice. We had to decide on whether the pattern matched with the theoretical propositions or it didn’t. Where the pattern was deemed to hold true then the theory would be tested in the positive, where it didn’t the theory would have failed to explain our case.

Methodological tactics – Data collection

A case study is an in-depth investigation that seeks to uncover the various nuances, patterns and latent elements that other research approaches may overlook (Berg 2004). It accordingly makes use of different methods to collect various kinds of empirical data (Hamel 1993). Our case study made use of the more traditional methods of collecting data. These were questionnaires, participant observation and the review of documents. Data collection through the participant observation and document review techniques is representative of the 12 year period. Participant observation was a crucial data gathering technique as all researchers have been involved with OASIS at some point over its lifetime. Documentation used in this research included support logs and records from the OASIS system. To capture current perceptions, questionnaires were sent to all current user and non-user groups of OASIS identified as academic staff.

At the time of the writing of this paper 94 responses (34.9%) had been received from users of OASIS and 18 responses (15.3%) from non-users. The analysis in this paper concentrates on the free text responses to two open questions asking staff about the factors influencing their perceptions of the usefulness and ease of use of OASIS. Academic staff members in charge of a course are responsible for making the OASIS adoption decision. Students are only able to use OASIS after this adoption decision is made. For this reason we have initially focused on academic staff as the users and non-users of OASIS. Subsequent research will investigate student perceptions.

Scientific value and study limitations

The case study is a popular research approach across many disciplines both basic and applied (Hamel 1993). Despite their popularity they have many strong critics due to the belief that the approach lacks insufficient objectivity and concern over the ability to generalize research results (Berg 2004). We have been careful in our research to maintain objectivity through deliberate construction of a research design. Construction of such a design does much to increase the rigor of a study and counter the claims of “weak research” (Yin 1994). As part of our research design we maintained as much objectivity as possible by having each researcher separately review the evidence as part of the data analysis phase. Our findings have been examined within the context of our chosen theoretical framework; TAM. As far as generalizing the results of this research is concerned our view is similar to that taken by Berg (2004). He states that “When case studies are properly undertaken, they should not only fit the specific event studied but also generally provide understanding about similar … events. The logic behind this has to do with the fact that few human behaviours are unique, idiosyncratic and spontaneous.” (Berg 2004). Likewise it is our belief, due to the socially constructed nature of IS in general, as well as their reliance on social aspects as determinants of success in particular, that the success of OASIS is not a unique event. Our study should therefore provide an understanding to the wider community of success in similar IS implementations.

Theoretical background

Success of an IS innovation can be determined in a number of ways. However, general organisational measures of success include “on time and on budget” (Standish Group 1995, IT Cortex 2002) with the desired functionality (Mahaney and Lederer 1999). Following the logic inherent within the literature focussing on IS characteristics predictive of success, failure is generally discussed in terms of having the opposite characteristics. For example, Whittaker (1999) described the 1997 KPMG survey on what constituted an IS project failure. The study deemed a project as having failed if it overran its budget by 30% or overran its schedule by 30%, or the project was cancelled or deferred due to non-delivery of planned benefits. However, Mahaney and Lederer (1999) argue that there are degrees of failure and that a project that overruns budget by 5% is less of a failure than one that overruns by 50%. Some determinants of why IS innovations might be considered failures include whether they; have the ability to evolve and grow with the organization, integrate well with the business environment, possess consistency between the initial requirements and the final solution, simply make business sense (IT Cortex 2002).

DeLone and McLean (1992) were leaders in moving to a more user centred approach when trying to judge overall IS success. Their model suggests six interdependent measurements of success; system quality, information quality, use, user satisfaction, individual impact and organisational impact. It is important to note that all of these factors should be considered when trying to measure success under the model and that no single measure is intrinsically better than any other. Further attempts have been made to refine and expand on their model by others (e.g. Seddon et al. 1999) as well as minor refinements suggested by themselves (DeLone and McLean 2003). However as DeLone and McLean (1992 p. 61) themselves point out, “there are nearly as many measures of success as there are studies”.

With the more recent study conducted by Iivari (2003) there is more evidence to suggest the applicability of the DeLone and McLean (1992) model in measuring a system’s success. This work helps to contribute to the shift from organisational measures of success to more user focused measures. Davis et al.’s (1989) work on the TAM IS theory, is a user centred approach which has gained popularity as a measure of a users acceptance of technology. We draw the conclusion that if a system enjoys high user acceptance this will impact positively on system use. Use of the system is a contributing factor to system success especially when that system is not mandatory (DeLone and McLean 2003, Iivari 2005). Based on this assumption we use TAM as a theoretical framework to guide our research. Specifically, do the constructs in the model offer a reasonable explanation for why OASIS has enjoyed such an exponential growth in its adoption and use?

System background

Online Assignment Submission, Infocom System (OASIS) arose out of early experiments in 1994 by a single academic implementing a system to reduce assignment turnaround for distance students (Jones and Jamieson, 1997). Adoption of OASIS by other academics was limited at this time. Only 13 course offerings made use of the system with just over 1900 assignments being submitted in the six years up until 2000. Since 2000 use of OASIS has increased significantly. From the years 2000 to 2005 over 77,000 assignments have been submitted via OASIS by 6892 (72+%)of Infocom students.

Students enrolled in Infocom courses are distributed across a number of campuses as well as being enrolled via distance education. There are five regional Central Queensland (CQ) campuses in Bundaberg, Emerald, Gladstone, Mackay and Rockhampton. Four other Australian International campuses (AICs) in Brisbane, Gold Coast, Sydney and Melbourne managed by a commercial partner. Campuses are also located overseas in Fiji, Singapore, Malaysia, Hong Kong and China. Students may also study from any location in the world via distance education (FLEX). Figure 1 provides a summary of Infocom student numbers from 1996 to 2005.

OASIS usage in percentage

Figure 1. Number and type of students enrolled in Infocom Courses (1996-2005)

Since its inception Infocom has had a small development team responsible for its online presence. In 2001, partly in response to increasing numbers, this team was expanded and additional effort placed on providing services that would help support Infocom’s teaching operations. Using an agile development methodology (Jones and Gregor, 2004) this group, in response to direct user feedback, made a range of additions to OASIS to improve its functionality. The combination of increasing complexity and this on-going development of OASIS appears to have had an impact on usage of OASIS. Figure 2 shows percentage of Infocom students, staff and courses using OASIS from 2000-2005. Specific staff figures are only available from 2002 onwards when a markers’ database was added to the system.

Number and type of students enrolled in Infocom Courses

Figure 2: OASIS usage in percentages of Infocom students, courses, and staff (* figures as of September 30, 2005)

Findings

In this study we applied the TAM measures of perceived usefulness and perceived ease of use to two different staff groups; non-users and users. Non-user perceptions concerning usefulness and ease of use of the system are used to compare against actual beliefs of users. The non-user group is examined due to their potential to become users and therefore impact on the continued growth of the system’s popularity. The user group is examined due to their importance in maintaining the current level of system use. This section reveals the complex underlying belief structures concerning the two constructs of perceived usefulness and ease of use as they pertain to OASIS. This is information that has been missed in other investigations concerning TAM (Segars and Grover 1993). Of particular interest it reveals that usefulness and ease of use, at least in our study, seem to be influenced positively by the evolutionary development model adopted by the system support team. This is in line with other research concerning what makes for a successful system, namely the ability of the system to evolve with the business (IT Cortex 2002). The findings in this study are of a preliminary nature only.

How do non-users perceive OASIS?

Overall non-users of OASIS had mainly positive perceptions of the system. These perceptions centred on the belief that the system would benefit the students just as much as it would enhance course management.

Perceived usefulness factors

The main student benefit perceived by non-users was timely turn-around of assignments. One respondent noted that ‘OASIS will eliminate unnecessary delays’ while another believed that ‘It may also help with prompt and efficient grade information requests in/out’. Other respondents saw OASIS as a method for improving courses by being able to more easily analyse the results from assessment. One respondent noted that ‘each question can be analysed for effectiveness at distinguishing between students passing and failing. With individual questions assessed for how well they are answered teaching can be modified to prepare students better in the identified weak area’. Another was the benefits in being able to track how well students were progressing, stating that ‘I envisage OASIS would be useful to gauge student progress/understanding/level of expertise throughout a particular course/subject’.

Non-users also perceived a number of administrative benefits from OASIS. Of particular note was the ability to track assignments and marking, with one respondent stating their belief that OASIS ‘will encompass safe guards for assignment delivery and return, as well as acknowledgement of assignment receipt for students’. This is an important aspect of course administration, especially with the difficulties in distributing, managing and moderating marking over multiple campuses and markers. Another respondent support this belief, stating that ‘I have used a similar system before and it was quite helpful to my consolidating marks, and not being on campus would probably simplify the marking system’. Respondents believed that OASIS would make this task both possible and easier. Others believed that OASIS would also provide additional benefit by enabling the use of automated plagiarism detection.

While there were few negative perceptions of the usefulness of OASIS, some respondents had beliefs about what types of assessment OASIS was suitable for. One noted that ‘I believe OASIS is suitable for multiple choice questions. But my assignments are essay type with computer program printouts. As of now, I don’t know how I can make OASIS useful for my course’. Another indicated that OASIS didn’t fit in with the way that they currently assessed, stating that ‘I mark all my student’s assignments manually, it is easier for me to sub-edit stories that way.’.

However, respondents generally had positive perceptions of OASIS, with one respondent stating that ‘certainly, any online submission technology would be useful to me. And the precedent of other IT systems made available in Infocom suggests that it would be extremely user friendly for people with very limited computer competence/confidence. The nifty acronym is also appealing’. The successful evolutionary development process adopted by the support team had produced a number of successful systems and helped in developing positive perceptions amongst users of new systems

Perceived ease of use factors

Non-user respondents generally believed that OASIS would be easy to use. They justified this with two belief factor groups. The first group was the technology centric belief that, as non-users had used similar systems, they would be able to easily use OASIS. One respondent noted that ‘I have been using computers for many years, including online application/enrolment. These may not be identical with OASIS, but I believe there will be similarities’. Another respondent stated that ‘It should not be difficult for me to learn since I’m computer literate’. Another respondent believed that the system would be just as easy to use as other systems developed by the faculty, stating that ‘my positive experience with other Infocom systems gives me confidence that OASIS would be no different. The systems team have a very good track record that inspires confidence’.

The second group of factors was based on having not heard negative things about ease of use of the system. One respondent noted that ‘nobody seem to complain too much about OASIS being hard to use, or hindering them in their job’. However, this was contradicted by another respondent who stated that ‘I have heard from another tutor that OASIS is a bit time consuming and a little confusing… but [I] have not used it myself’.

How do users perceive OASIS?

Users of OASIS had generally positive perceptions of the system. Again, these perceptions centred on the belief that the system would benefit the students just as much as it would enhance course management. Yet users were pragmatic in their beliefs and many discussed the “trade-offs” associated with using OASIS. However, a new category of perceived usefulness was uncovered concerning the personal benefits of using the system.

Perceived usefulness factors

Users believed that OASIS gave them a greater ability to monitor student progression while also allowing students to track their assessment through the marking process. One respondent noted that ‘submission records for students are useful in monitoring my students’ progress, hence adjust tutorials/support as needed’. Another respondent supported this by stating ‘it is easy (and quicker) to know if a student has submitted work for assessment by checking the relevant section of the web site’. Others saw the advantage in being able to compare and contrast assessment results, highly rating OASIS’s functionality to give teaching staff the ‘ability to compare your student’s results with overall performance’. Many staff also saw the non-repudiation aspects of the assignment management as being advantageous, with one user stating ‘students cannot say that they were NOT late or did submit the assignment (when in fact they did not)’.

Administratively, users discussed a number of factors that they perceived made OASIS useful. Most of these concerned assignment management issues. One issue identified was the ability for the user to track where an assignment was and what actions had been performed on it, with one respondent noting that ‘[OASIS] makes assignment collection simple and easy also [as you] do not have assignments go missing. [OASIS is a] quick and easy way of returning assignments and collect the assignment marks’. This tracking also facilitates moderation processes, and as noted by another respondent, ‘OASIS allows for the moderation process to be carried out in a timely fashion”. OASIS was also seen to support core academic requirements as exemplified by one user who stated that ‘OASIS is useful in the case of essay and report type assignments as it helps in detecting plagiarism’. This issue is especially important over a multi-campus operation, with another respondent adding that one of the key usefulness factors was the ability of OASIS to ‘perform copy detection, not only within a campus but between campuses’

The users of OASIS also found a personal usefulness factor in the ability to remotely download assessment to mark and moderate. This gave them the ability to mark, moderate and manage assessment from anywhere in the world. One respondent stated that having ‘assignments on soft-copy [was] a tremendous help [because there was] no need to carry them home’. It was sessional staff who seemed to gain the most benefit from electronic access to assessment. One user explained: ‘I am sessional lecturer and OASIS makes it possible for me to download the assignments for marking. I don’t have to go to the campus to get the submitted assignments’. Another supported this by saying that ‘OASIS is useful because it has enabled me to work from home and pick up students’ assignments outside office hours’.

The most contentious usefulness factor was the benefit OASIS was able to provide in the time taken to mark assessment. While many users found the systems fast and efficient to use, others disagreed, but continued to use the system because of other usefulness factors. One user who found that OASIS saved time stated that:

Having also been a marker (both paper-based, and using OASIS), I was stunned by just how much time was saved by no longer needing to handle piles of paper. Virus scanning 100 floppy disks, for example, takes a long time. OASIS provides a neatly formatted, scanned, and correctly-named set of files

However, several users found the process of dealing with electronic assignments time consuming and cumbersome. One respondent stated that ‘practical experience with many assignments that were to be submitted through the OASIS system indicates that … it takes much more time and effort to mark assignments on line’ while another noted that ‘for assignments that where marking can not be automated it is very time consuming to mark electronic copy, especially when there is significant reading to be done. it is also time consuming to provide feedback’. Many staff made comparisons with OASIS and “hard-copy” marking, with one user stating that:

It is a very good way of submission and collection of assignment. But, the hardest part of it, is adding comments electronically during the marking process. It kills time. I did same types of marking to some other university, and later they decided to take hardcopies and to write comments. We found that saving 50% of the overall marking time.

The general experience of users of OASIS was that it was initially slow to mark assessment with, but as one user who stated that ‘[marking with OASIS] takes a little longer to generate a rhythm to freely mark assignments in an efficient timeframe’. Many noted that the time to mark assessment was dependent of the type and complexity of assessment, with one user stating ‘the experience is very dependent on the assessment design’. Even with these negative aspects, users still perceived it as useful with one user summarising by stating that ‘it’s a great system but online marking and commenting takes significantly longer than on hard copy – other than that I like its functionality’.

The only other negative factors that affected perceived usefulness were those concerning support. Some users felt that OASIS was complicated, difficult to understand and lack support mechanisms. One user’s frustration was evident from the comment ‘just trying to understand how to use [it] is a pain’. Yet other users made particular mention of the support services offered by the web team. This perception of the support services will be discussed later as a factor concerning ease of use.

Perceived ease of use factors

Users generally perceived OASIS as easy to use, however two factor groups, technology and support, affected these beliefs. Technology affected users in both positive and negative ways. Users who were comfortable with technology believed OASIS was easy to use and made comments such as ‘being an IT professional, I find it very very easy to interact with’. However, users who found technology intimidating focused on this as impeding their use of OASIS. One user remarked that ‘anything computer-mediated the comments relating to ease of use and technology were focused on difficulties with understanding technology external to OASIS rather than the system itself

A related group of factors concerned support mechanisms. As previously discussed, the perception of a lack of support mechanisms impacted negatively on the perceived usefulness. However, users of the system were divided on the issue. In particular, those users who had used OASIS over a long period of time and had watched its support mechanisms evolve saw them as a positive influence on ease of use. One respondent supported this by stating that ‘It used to be a problem, but I’ve seen the system and supporting documentation improve to the point that I would consider the system fairly easy to use for new users. Support requests for the system have dropped significantly as it has matured’. Another remarked that ‘OASIS is self explaining, there is not much to learn about it in order to use it’. However, others still regarded the system as difficult to initially learn, with one noting that ‘learning OASIS for the first time is difficult because the instructions are not very clear. However, it is easy once you get the hang of it’. Others noted that the lack of documentation and online help procedures was overcome with support from the web team, with one respondent commenting that ‘learning how to do things in the system is not easy but the tech team offer an excellent support and are to be commended for their efforts’. Most users shared these views believing that once users started using OASIS, the perception of ease of use changed. This was supported by one user who stated that ‘OASIS is no more difficult or easier to use than any other web-based system with online help and hyperlinks to the various relevant parts. I think initially I asked colleagues about its general use as the concept seemed daunting at the time (before I’d actually used it)’.

Discussion and conclusion

In general users had very positive perceptions surrounding the usefulness of OASIS. As one respondent stated, ‘I find the system professional and bug free. It’s an excellent assignment management tool. It provides a rigid framework for student submissions. Students appear to have little or no problem with the general concept of online submission and its use. In all I find the system very useful’. If system success can of this system by the majority of students in the Faculty of Informatics and Communication at CQU and its use by non-mandatory nature of the system giving even more strength to the motivations behind its adoption. In the search for an explanation of why it has been so successful we applied the TAM to investigate both staff users and nonusers of the system. Both of these groups revealed very positive perceptions and beliefs surrounding the usefulness and ease of use constructs in TAM. On further investigation these constructs were complex in nature but seemed to centre more on the administrative benefits that the system could provide rather than the pedagogical benefits originally intended by use of the system.

In examining non-users perceptions and users’ beliefs, we have presented evidence that provides an explanation for the continued and growing success of OASIS. Non-users perceive that the system will be useful and easy to use and will not hesitate in using it when the chance arises. This indicates why the believe that the system is useful and easy to use and this explains stability in growth and continued use. If success can be measured in terms of use then we believe that the usefulness and ease of use factors within TAM are reasonable predictors of system success. We also believe that the usefulness and ease of use constructs are positively influenced by the successful application of agile development methods employed by the support staff of the system. As shown above this process has generated a perception amongst staff that the systems produced by this team will be useful and easy to use, or if the systems are not useful or easy to use that the predict that the success of OASIS will continue as long as the beliefs and perceptions concerning the system’s usefulness and easy to use characteristics are maintained through activities such as evolutionary development.

As stated in the previous section, methodological tactics, our findings are limited to initial analysis of two free text questions. Further detailed analysis has to be carried out on all questions. It is envisaged that in the information of respondents. This may impact on their perceptions of perceived usefulness and perceived ease of use of the system. Although the preliminary results of this study offers a more detailed account of a specific systems success it would benefit from expansion in several areas. Firstly, expanding the study to include students. Secondly, using UTAUT instead of TAM as the theoretical framework. Thirdly, including other cases to see whether the results still hold. Finally it may be useful to investigate further the complex structure of the perceived ease of use and perceived usefulness constructs of TAM to other information systems supporting teaching and learning as well as other more complex information system innovations.

References

Adams, D. A., Nelson, R. R. & Todd, P. A. (1992) Perceived usefulness, ease of use, and usage of information technology: A replication. MIS Quarterly, 16, 227-247.

Bagozzi, R. P., Davis, F. D. & Warshaw, P. R. (1992) Development and Test of a Theory of Technological Learning and Usage. Human Relations, 45.

Berg, B. L. (2004) Qualitative Research Methods for the Social Sciences, Boston, Pearson Education, Inc.

Davis, F. D. (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319-340.

Davis, F. D., Bagozzi, R. P. & Warshaw, P. R. (1989) User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35, 982-1003.

Delone, W. H. & Mclean, E. R. (1992) Information systems success: The quest for the dependent variable. Information Systems Research, 3, 60-95.

Delone, W. H. & Mclean, E. R. (2003) The DeLone and McClean Model of Information Systems Success: A Ten-Year Update. Journal of Management Information Systems, 19, 9-30.

Denscombe, M. (1998) The Good Research Guide for small-scale social research projects,
Philadelphia, Open University Press.

Hamel, J. (1993) Case Study Methods, Newbury Park, SAGE Publications.

Hendrickson, A. R., Massey, P. D. & Warshaw, P. R. (1989) On the the test-retest reliability of perceived usefulness and perceived ease of use scales. MIS Quarterly, 17, 227-230.

Iivari, J. (2005) An Empirical Test of the DeLone-McLean Model of Information System Success. The DATA BASE for Advances in Information Systems, 36, 8-18.

IT Cortex (2002) Success assessment: Where is the limit between success and failure? Chapelier, Jean-Pol.

Jones, D. & Behrens, S. (2003) Online Assignment Management: An Evolutionary Tale. 36th Annual Hawaii International Conference on System Sciences. Hawaii, IEEE.

Jones, D. & Buchanan, R. (1996) The design of an integrated online learning environment. IN Allan Christie, P. J., Beverley Vaughan (Ed.) Proceedings of ASCILITE’96. Adelaide.

Jones, D. & Jamieson, B. (1997) Three Generations of Online Assignment Management. IN Kevill, R., Oliver, R. & Phillips, R. (Eds.) ASCILITE’97. Perth, Australia.

Jones, D., Lynch, T. & Jamieson, K. (2003) Emergent Development of Web-based Education. Proceedings of Informing Science + IT Education. Pori, Finland.

Mahaney, R. C. & Lederer, A. L. (1999) Runaway information systems projects and escalating commitment. Special Interest Group on Computer Personnel Research Annual Conference. New Orleans, Lousiana, USA, ACM Press.

Seddon, P. B., Staples, S., Patnayakuni, R. & Bowtell, M. (1999) Dimensions of information systems success. Communications of the AIS, 2.

Segars, A. H. & Grover, V. (1993) Re-examining perceived ease of use and usefulness: A confirmatory factor analysis. MIS Quarterly, 17, 517-525.

Stake, R. C. (1995) The art of case study research, Thousand Oaks, CA, Sage.

Standish Group (1995) The CHAOS report. The Standish Group International.

Subramanian, G. H. (1994) A replication of perceived usefulness and perceived ease of use measurement. Decision Sciences, 25, 863-873.

Szajna, B. (1994) Software Evaluation and choice: predictive evaluation of the Technology Acceptance Instrument. MIS Quarterly, 18, 319-324.

Whittaker, B. (1999) What went wrong? Unsuccessful information technology projects. Information Management & Computer Security, 7, 23-30.

Yin, R. K. (1994) Case Study Research Design and Methods, Thousand Oaks, SAGE Publications.