Between the idea and the reality,…. falls the shadow

T. S. Elliot’s poem The Hollow Men has a section that goes

Between the idea
And the reality
Between the motion
And the act
Falls the Shadow

I find some connection with this section due to my interest in shadow systems. Applications of information technology that arise within organisations without the knowledge of, and often in spite of, the official, centralised information technology division.

The idea

The typical response of organisational officialdom is that shadow systems are horrible, beastly things that must be stamped out. These horribly inefficient, unscalable, incorrect toys cobbled together by amateurs create problems for the organisation with no corresponding benefit. All IT development must be done by the heroic, rigorous and knowledgeable folk within the central IT division. This is the idea that underpins much of the existing IT development practice within organisations.

The content of the current wikipedia page on shadow systems gives some insight into this view. The wikipedia page has three sections:

  1. Overview – general summary of shadow systems and what they are.
  2. Cause – some general points about what leads to the development of shadow systems.
  3. Problems – a larger list of the problems shadow systems exhibit. Including, poorly designed, not scalable, poorly documented, untested, may allow unauthorised access to information, easy to introduce errors, one hard disk failure away from disaster, and several versions of the truth.

There is no mention of any potential benefits that shadow systems may provide. One potential benefit that is somewhat mentioned gets twisted. Often shadow systems will enable the real work of the organisation to go ahead in spite of unresponsive IT systems. This gets a mention but in a negative way as a cause of shadow systems, not as a benefit. The description of this “benefit” on the page places shadow systems in a negative light and uncritically accentuate the positives of centralised IT divisions. For example, (emphasis added)

Quite properly, when a reporting system is put together by IT professionals, they need to consider all aspects of how the system will be used.

The various skills that are required to achieve all of this means that inevitably a number of different people will all be involved in the task of creating the new report. This increases the amount of time and effort it takes to put a rigorously engineered solution in place. Shadow Systems typically ignore this kind of rigor, making them much faster to implement, but less reliable and more difficult to maintain.

I find it particularly interesting that the current Wikipedia page on shadow systems seems to contain much of the same content that appears on a commercial website that is linked to from the Wikipedia page. I find it increasingly interesting that the purpose of the commercial website is promote the idea of business intelligence applications and if you dig a bit further is associated with a company selling “ERP” solutions.

The reality

Behrens and Sedera (2004) developed a model (shown below) in an attempt to explain why shadow systems exist.

Sandy's Shadow System Model

The model suggests that shadow systems are created by a gap between the needs of organisation and its use of information technology and the features provided by the organisational IT systems. This mismatch can arise due to characteristics of the people, business processes, organisation and technology. How the gap is overcome is mediated by the available resources and support.

The reality is that the organisational information technology resources are not providing the features required. They are failing.

The shadow

The shadow system becomes the method by which the people within the organisation overcome this gap. How they attempt to address the failing of the organisational information technology division to provide what is required by the organisation.

Overcoming the negative perception of shadow systems

Members of the central IT divisions really hate shadow systems because, at least sub-conciously, they are aware that shadow systems point out where they have failed. And no-one likes having their failures pointed out. Especially folk within central IT divisions who generally have a pretty tough time of it. Especially because most of them have this image of themselves as the supreme, rigorous rationalists. They believe they can and do spend sufficient time and intelligence analysing the situation and coming up with the best solution for the organisation.

What they don’t realise is that there is never going to be one best solution. The implementation and use of information technology within any sufficiently large and complex organisation is going to be a wicked problem and one of the defining characteristics of a wicked problem is that there is no one best solution.

There appears to be some value for IT and broader organisational management to overcome this automatic and negative reaction to shadow systems and learn that they can help identify, and perhaps, solve organisational failures by valuing and seeking to understand shadow systems and the gap between idea and reality.

Perhaps rather than seeking to squash shadow systems, IT divisions should seek to encourage the development of shadow systems and put in place practices and support which mitigate some of the flaws of shadow systems. Such a practice might leverage the increasing capability of the current generation of end-user tools that make “shadow system” development so easy and also seek to address the problem of reducing costs most IT divisions seek.

There is a great chance that shadow systems are almost certainly going to be the source of more innovation than central IT divisions. This is because there will be more diversity of perspective around the development of shadow systems than exist within the development arms of central IT divisions. And diversity and through it perspective shift is a necessary condition for innovation.

References

Sandy Behrens, Wasana Sedera, Why do Shadow Systems Exist after an ERP Implementation? Lessons from a Case Study, Proceedings of PACIS’2004, Shanghai, China

3 thoughts on “Between the idea and the reality,…. falls the shadow

  1. matsonian

    Having an idea is one step, implementing it is the next… or is it? How about developing a plan for implementation like that which is developed using the QuickPlanner Plus method of planning. QPP helps the individual to develop idea from thought to fruition by helping to define the goals, objectives and vision.

    http://www.quickplannerplus.com

  2. G’day Matson,

    I tossed and turned a bit about whether or not I would spam your comment.

    There were three main reasons I was going to spam it
    * You don’t mention that quickplanner plus is your company
    * My use of “idea” in this post wasn’t intended in the same way you used it
    * The over-emphasis on “planning” is one of the major flaws contributing to shadow systems

    Your company

    Your blogger profile (http://www.blogger.com/profile/14628159963370941143) lists the quickplannerplus website as “My Website”

    The idea

    IN the above I intended “the idea” to represent the idea held be management and the central IT division about what was happening in the organisation around the use of their IT systems and the costs/benefits of those systems.

    For me, the idea is separated from the reality of organisational life by a gap. A gap that is bridged by shadow systems.

    The main idea of this post is that central IT divisions see shadow systems as something to trashed out of hand, rather than understood as signifying that there is a gap and that perhaps they should learn about the causes of that gap and rectify the idea to be closer to the reality.

    Over emphasis on plan-driven approaches.

    A major contributor, IMHO, to the gap between the idea and reality is the on-going, over use of plan-driven approaches by central IT divisions and senior management. I haven’t looked in detail but it appears that quickplannerplus assumes and enhances this over use of plan-driven approaches.

    I’ve talked about the problems of these approaches and the alternatives in other blogs posts and in the other publications.

    e.g.
    https://davidtjones.wordpress.com/publications/the-teleological-brake-on-icts-in-open-and-distance-learning/
    https://davidtjones.wordpress.com/publications/the-teleological-reason-why-icts-limit-choice-for-university-learners-and-learning/

    In the end, I decided to allow your post to provide a platform to repeat my platform and maybe generate some additional debate.

    David.

  3. Pingback: The protean nature of modern technology - another limitation of most views of e-learning « The Weblog of (a) David Jones

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s