the introduction of new media is a special historical occasion when patterns anchored in older media that have provided the stable currency for social exchange are reexamined, challenged, and defended.
In that previous post I applied this idea to e-learning. In this post I’d like to apply this idea to academic research.
In this post Jon Udell talks about the dissonance between the nature of blogs, the narrative form he recommends for blogs and the practices of academics. In it he quotes an academic’s response to his ideas for writing blogs as
I wouldn’t want to publish a half-baked idea.
Jon closes the blog post with the following paragraph
That outcome left me wondering again about the tradeoffs between academia’s longer cycles and the blogosphere’s shorter ones. Granting that these are complementary modes, does blogging exemplify agile methods — advance in small increments, test continuously, release early and often — that academia could use more of? That’s my half-baked thought for today.
I think this perspective sums it up nicely. The patterns of use around the old/current media for academic research (conference and journal papers) are similar to heavyweight software development methodologies. They rely on a lot of up-front analysis and design to ensure that the solution is 100% okay. While the patterns of use of the blogosphere is very much more like that of agile development methods. Small changes, get it working, get it out and learn from that experience to inform the next small change.
There are many other examples of this, just two include:
- Dave Snowden’s distinction between fail-safe design (old style) and safe-fail design probes (new style).
- Lucas Introna’s (1996) distinction between teleological design (old style) and ateleological design (new style).
Essentially the standard practices associated with research projects in academia prevent many folk from engaging in getting the “half-baked ideas” out into the blogosphere. There are a number of reasons, but most come back to not looking like a fool. I’ve seen this many times with my colleagues wanting to spend vast amounts of time completing a blog post.
As a strong proponent and promoter of ateleological design processes, I’m interested in how this could be incorporated into research. Yesterday, in discussions with a colleague, I think we decided to give it a go.
What we’re doing and what is the problem?
How do you data mine/evaluate usage statistics from the logs and databases of a learning management system to draw useful conclusions about student learning, or the success or otherwise of these systems.
This is not a new set of questions. The data mining of such logs is quite a common practice and has a collection of approaches and publications. So, the questions for use become:
- How can we contribute or do something different than what already exists?
- How can we ensure that what we do is interesting and correct?
- How do we effectively identify the limitations and holes underpinning existing work and our own work?
The traditional approach would be for us (or at least Col) to go away, read all the literature, do a lot of thinking and come up with some ideas that are tested. The drawback of this approach is that there is limited input from other people with different perspectives. A few friends and colleagues of Col’s might get involved during the process, however, most of the feedback comes at the end when he’s published (or trying to publish) the work.
This might be too late. Is there a way to get more feedback earlier? To implement Udell’s idea of release early and release often?
Safe-fail probes as a basis for research
The nature of the indicators project is that there will be a lot of exploration to see if there are interesting metrics/analyses that can be done on the logs to establish useful KPIs, measurements etc. Some will work, some won’t and some will be fundamentally flawed from a statistical, learning or some other perspective.
So rather than do all this “internally” I suggested to Col that we blog any and all of the indicators we try and then encourage a broad array of folk to examine and discuss what was found. Hopefully generate some input that will take the project in new and interesting directions.
Col’s already started this process with the latest post on his blog.
In thinking about this I can come up with at least two major problems to overcome:
- How to encourage a sufficient number and diversity of people to read the blog posts and contribute?
People are busy. Especially where we are. My initial suggestion is that it would be best if the people commenting on these posts included expertise in: statistics; instructional design (or associated areas); a couple of “coal-face” academics of varying backgrounds, approaches and disciplines; a senior manager or two; and some other researchers within this area. Not an easy group to get together!
- How to enable that diversity of folk to understand what we’re doing and for us to understand what they’re getting at?
By its nature this type of work draws on a range of different expertise. Each expert will bring a different set of perspectives and will typically assume everyone is aware of them. We won’t be. How do you keep all this at a level that everyone can effectively share their perspectives?
For example, I’m not sure I fully understand all of the details of the couple of metrics Col has talked about in his recent post. This makes it very difficult to comment on the metrics and re-create them.
Overcoming these problems, in itself, is probably a worthwhile activity. It could establish a broader network of contacts that may prove useful in the longer term. It would also require that the people sharing perspectives on the indicators would gain experience in crafting their writing in a way that maximises understandability by others.
If we’re able to overcome these two problems it should produce a lot of discussion and ideas that contributes to new approaches to this type of work and also to publications.
Outstanding questions include:
- What are the potential drawbacks of this idea?
The main fear I guess of folk is that someone, not directly involved in the discussion, steals the ideas and publishes them unattributed and before we can publish. There’s probably a chance that we’ll also look like fools.
- How do you attribute ideas and handle authorship of publications?
If a bunch of folk contribute good ideas which we incorporate and then publish, should they be co-authors, simply referenced appropriately, or something else? Should it be a case by case basis with a lot of up-front discussion?
- How should it be done?
Should we simply post to our blogs and invite people to participate and comment on the blogs? Should we make use of some of the ideas Col has identified around learning networks? For example, agree on common tags for blog posts and del.icio.us etc. Provide a central point to bring all this together?
Lucas Introna. (1996) Notes on ateleological information systems development, Information Technology & People. 9(4): 20-39