Exploring Moodle Book usage – Part 6 – What do they contain?

Part 6 of this series diverges a bit from the last post and moves away from what people are doing with the Book resources to focus on the contents of the Book resources themselves.  Questions I’m hoping to explore in this post include:

  • How long are the Book resources?
    Measured perhaps in number of chapters, bytes, and perhaps textual word count.
  • Are the Book’s web or print documents?
    Do they include links? To other books in the course? To external sites? Which sites? Do they include multimedia?
  • What does one book with 500+ links actually link to?
  • How readable is the text?

NOTE: Click on the graphs below to see larger versions.

How long are the Book resources

A Moodle Book resource is a collection of “chapters” and “sub-chapters”, which are essentially web pages. The following starts looking in more detail at these chapters and their contents.

Where did they come from – import or create?

Looking more closely at the chapters provides an opportunity to find out how they were created.

Each chapter has a field importsrc which specifies the name of a file from which the content was imported.  Indicating that the chapter was created by uploading a already written file, rather than using the Book online editing interface.

Analysis shows that only

  • 9.8% (2397 out of 24408) of chapters are imported;
  • these belong to 10.2% (287 out of 2801) of books; and,
  • 11.8% (44 out of 374) of courses.

i.e. ~ 90% of chapters, books and courses are created by using the online Book interface.  Not a great way to create.

How many chapters per book?

The next step is to have a look at how long each book is based on the number of chapters. This isn’t a great indication of length because each chapter is simply a web page, it could be quite short, or quite long.

The following graph shows the number of chapters in every book grouped by year. Overall the number of chapters stays pretty much the same.  However, there are a couple of strange outliers tending toward 100 chapters in a book. The median number of chapters per book has increased from 6 in 2012 to 8 in 2015.

chapters per book per year

The total number of books shown in the above graph for each year is a bit out from earlier data. I will need to come back to these analysis and nail down what courses/books are counted in each analysis.

How many words in each book?

To get a better idea of the size of books the aim here is to convert the chapter content to plain text and do some analysis of the text.  This is where the beauty of Perl (confirmation bias) comes to the fore.  There’s a module for that.

The following graph maps the number of words for each book by year.  It shows that in 2014 and 2015 the number of words per chapter/book was certainly getting longer.  The median went from 1157 words per book to 1718 per book (with a dip in 2013 back to 1004 words per book). The upper limit moved from 5282 words in a book to 6930 words per book. Scarily, there are outlier books that are approaching (and in some cases bypassing) 60,000 words in length.

To give you some idea of read time, I’ll use Medium’s method for calculating read time (ignoring images) to convert the numbers into minutes to read:

  • Around the median word count – 1700 words – equates to about 6.1 minutes.
  • The maximum upper word count – 6930 words – equates to about 25.2 minutes.
  • The outliers – around 60,000 words – equates to about 218.2 minutes, which is approaching 4 hours.

Adding to this is that I’m not sure the typography and design of your typical Moodle Book is going to match what you might expect on Medium. Not to mention that Medium don’t mention if their average adult reading spead (275 words per minute) is for words on print or screen.

words per book per year

Readability?

The module that calculates words also does readability tests, including the Flesch reading-ease test. The following graph shows the results on that test for each of the books grouped by year.

Grain of salt – The graph does exclude a number of books that achieved negative results on the test. Initially, it appears that this may be due to the conversion to text only not handling some special characters which worsen the readability.  (Apparently it is possible to get a negative value on the test). This may also be decreasing the “reading ease” of other books.  This will be examined more closely later.  But then again, quoting Wikipedia

While Amazon calculates the text of Moby Dick as 57.9,[9] one particularly long sentence about sharks in chapter 64 has a readability score of −146.77.

The median moves between 43.7 and 47.3, which is apparently around the 45 that Florida law requires for life insurance policy (thank you again Wikipedia).  However, the lower bound loiters around 5 suggesting very difficulty to read.  Wikipedia suggestions 30 to 50 as being the range for “college” and being difficult to read.

flesch per book per year

And my books?

Which has me wondering about mine. I think I’ve developed a tendency to reading difficulty.  The following graph shows the distribution for the latest offering of my main course that is contained in the data set.

That’s a nice-ish surprise.  Median at 60. Worst is 40 and best is 77. With better than 75% of the books above 50 which is the lower bound of the 10th  to 12th grade boundary.

However, I believe these results may be a little padded by the fact that I write most of my books in straight HTML. Meaning there’s no increase in complexity because of the difficulty of converting it into clean text.
EDC3100 S2 2015 readability
Which has me wondering about the evolution of readability.  The following graph shows the results from all offerings of the course that use the Book. A bit of a dip at the start with a small upward trend over time.  Not bad – but then of limited use given the limitations of this type of thing.

edc3100 readability through the ages

What about links – links per book?

One of the questions I’d like to answer is whether or not the people using the Book are using it as a poor-man’s replacement for a collection of paper, and how many are using it as a collection of web pages.  First exploration of this question is the rough indicator of how many links per book?

The following graph shows the number of links per Book per year. “Link” is defined here as any type of link, excluding a link to a style sheet. That means links to images, youtube videos etc are all counted as links.

As the graph shows there are a large number of books that have no links.  The median number of links is increasing each year. Starting at 11 in 2012 and moving through 13, 14, and finally 17 in 2015.  As the graph shows there are some major outliers with some Books having hundreds of links, including some with over 500 links.  These might include some of the very long books included above, but it might also include other books that contain huge numbers of links

In terms of books with very few links in 2012, 15.4% of the books had less then 3 links (remember that includes images, links, embedded videos etc) with 2014 having 16.1% and 2015 having 15.3%

num links per book per year

Links per book in EDC3100?

For a quick comparison, the following graph shows the number of links per Book for EDC3100 (the main course I use the Book in). Over time I  have been trying explicitly to think of the Book resources as collections of web pages.

The median # of links per book for all courses moved from 11 to 17. In EDC3100, the median has moved from 14 at its lowest (2013 S2 – a bad semester for links) up to 30 in 2015 (both semesters).  Similarly, the upper range for all courses ranged from 46 to 74 (driven by some truly large link numbers), for EDC3100 the upper range went from 43 in (2013 S2) up to 111 in 2015.

EDC3100 books links

Exploring types of links a bit more

The above couple of link graphs are limited because I really haven’t yet explored the diversity of link types that are included.  I had removed CSS links, but not script links.  I also haven’t split apart the different types of links. An examination which might shed some light on those strange books with 500+ links. Time than to explore.

Will try to identify the different types of links, generate stats for all the types, but when counting links, limit to more standard types (img/a)

Types of link to exclude from the count of links: iframe, embed, object, meta – handle link better.

The presence of <tag meta=”generator” looks like being one way of identifying chapters coming from Word.

Cleaning up the links does bring the numbers down a bit. e.g. the media for 2015 goes from 17 to 15, but the other medians stay the same. The upper for 2013 onward comes down by 1 to 3.

What about the 500+ books? What are those links

I’m interested in the books that have 500+ links.  What are they linking to?

One with 517 links has 510 <a links and 7 <img links. What are those 517 <a links?

Lots of internal links and all sorts of other links – other book chapters, readings. Looks like it might be a large book, is it?

29 chapters and 32,871 words – so a big, all in one book.

Exploring Moodle Book usage – Part 5 – more staff and student use

Continuing the exploration of how the Moodle Book module is being used, this post picks up from the last and will

  • Revisit the who is updating/creating posts, including data from the second half of 2015.
  • Explore the balance of all actions (print/view/update) by staff.
  • Explore the balance of all actions by students.

Who is updating/creating posts

The last post included a graph that showed generally (apart from two course offerings) that the core teaching staff appear to be doing the creation of books.  That graph had a few problems, including

  • Limited data from the 2nd half of 2015.
    Due to the switch in how Moodle logged events.  Need to handle the new log format.
  • Didn’t handle all roles.
    Appears there are some non-standard Moodle roles that the previous query didn’t handle.
  • Handling deleted books and chapters.
    I believe this is an issue for the new logging process which has connections back into the book and book chapters table. Which works nicely until books/chapters are deleted.

With those changes fixed, the following graph emerges show how many times each of the roles updated a Book resource in every course.  The changes between the following and the same graph in the last post, includes:

  • Significant increase in the number of updates for most roles (e.g. examiner up from 21968 to 31343; assistant examiner has almost doubled from 5144 to 10708)
  • Addition of the UNKNOWN role not in the previous graph

It should be noted that the following graphs do not include ~20K updates that I did in one course in one semester.

All book updates by role

And I thought it would be interesting to break down the updates by year to see what if there was any growth. Given the growth in the number of courses using the Book (17 in 2012 to 152 in 2015) there should always have been some growth.

Book updates by role by year

The graph above shows examiners making 2152 updates in 2012 and 13649 in 2015.  That’s a 6.3 times growth in number of updates for 12.6 times growth in the number of updates. Or, alternatively in 2012 a course examiner (on average) made 179 updates. In 2015 a course examiner (on average) made 90 updates.

Suggesting that the examiners are making less updates. Perhaps farming out the updating to other staff. The growth in edits by moderator and assistant examiner roles in 2014 and 2015 suggest that.  But more exploration is required.

Role balance of actions

Updating/creating is not the only action that can be done with a Book, you can also view and print parts or all of a Book resource. This step aims to explore what balance of actions each of the roles are involved with

For this purpose I’ve grouped log events into the following actions someone can perform on a Book

  • view – view a chapter or the entire book online
  • print – print a chapter or entire book
  • modify – delete or update a chapter/book
  • create – create or add a chapter or book
  • export – use the export to IMS option

The above updating/creating graphs including both modify and create actions.

The table shows the total events on all books by all roles from 2012 through 2015. It shows how viewing the book is by far the most prevalent action, accounting for 97.6% of actions.

Interestingly, at least for me, is that the percentage of modifications (1.1%) exceeds the percentage for printing (0.9%). I assume this is due to my outlier behaviour in 2015 in modifying a huge number.  Indeed it does.  The numbers in brackets in the table indicate a recalculation taking out that outlier.

Action # actions %
View 5040285 97.6  (98)
Print 46162 0.9 (0.9)
Modify 56754 (35867) 1.1 (0.7)
Create 18537 0.4 (0.4)
Export 1 1.9373E-05

Given the preponderance of viewing, the graphs tends to be a little less than useful by role. But the following look at usage by students and examiners.

 

Student usage

The graph below shows the spread of actions by students with the books. It shows that the most common action performed by students is viewing books. The table following the graph provides the raw data for the graph.

Student actions by year

Both this table and the one below for examiners show no print actions.  This suggests a bug in the analysis.

Another interesting point is the dip in printing between 2014 and 2015.  Even though the number of courses using books, and the number of views by students on books increased from 2014 to 2015, the number of print actions dropped. I wonder if this has anything to do with the large number of modify/create actions by students in 2015. Were the students creating the books/books created by students less likely to be printed?

Year View Print Modify Create
2012 386101 41 2
2013 812133 4487
2014 1447190 20310
2015 1967047 15198 1335 28

 

Examiner usage

The graph below shows the spread of actions by examiners with the books. The table following the graph provides the raw data for the graph.

The relative increase of modify/create actions by examiners between 2014 and 2015 is another indication of the 20000 updates I performed in 2015.

Examiner actions by year

The views and prints by examiners drop between 2014 and 2015

Year View Print Modify Create
2012 7193 2072 80
2013 26774 105 4850 495
2014 35855 647 8364 1833
2015 35185 452 26790 7746

 

Further questions to explore

  • What are the UNKNOWN roles?
  • How are the updates and other actions shown above distributed between users? Are there a small number of users making up the lion share of the actions (e.g. me and updates in 2015; and the one or two courses that had students updating books).
  • How many chapters do each student read? What about printing? Do they print and read online?
  • What is happening with print actions in 2012? Was there really no-one printing books?
  • Were the books created by students less likely to be printed? Did this account for the drop in print actions by students between 2014 and 2015? If not, what did?
  • Remove my 2015 outlier actions from the examiner actions graph and see what changes are made.

Exploring frameworks to understand OER/OEP

Some colleagues and are re-starting an exploration of OEP in Initial Teacher Education (ITE). A first task is an attempt to get a handle on what has been done/is known about OEP/OER. Yes, we’re looking for spectrums/frameworks/models etc that help map out what might be done with OEP/OER.  We’re interested in using this to understand what’s been done around OEP within ITE and also what we’ve already done.

The following is a summary of a quick lit review. No real structure and includes a range of strange notes.

OER adoption: a continuum for practice

Stagg (2014) offers the following continuum of practice

The proposed model seeks to acknowledge the complexity of applied knowledge required to fulsomely engage with open education by examining practitioner behaviours and the necessary supporting mechanisms. This conceptual model aims to be of use to both practitioners and also those responsible for designing professional development in an educational setting.

A continuum of practice - OEP

A Google Scholar search reveals some use this continuum.

Including Falconer et al (2016), which includes

We view our fourth category, enhancing pedagogy, as fundamentally different to that of producing high quality materials efficiently or cost effectively, in that it is underpinned by altruistic positions rather than a business model approach. It puts its emphasis on the value of the OER development process, rather than on the value of the OER content produced. (p. 99)

Through our analysis, some fundamental tensions have become apparent that will need to be resolved if the purposes of OER release are to be realised. (p. 101)

This limits imposed by a reputation-building motive are exacerbated at present as higher education institutions are encouraged to become increasingly competitive, elevating the importance of brand recognition. The consequence is a move away from risk-taking, towards a demand for predictable quality outcomes. This discourages innovation unless direct benefits can be proven in terms of new markets, student numbers, or shared costs of development and teaching. The benefits of OER in terms of institutional showcasing and attracting potential students, may prove attractive to institutional managers and gain institutional support for OER, but unless culture changes, they place inherent limitations on efficiency gains and the adoption of more open practices which are ultimately founded on a commitment to academic commons. (p. 102)

And develops some frameworks/continuums

Framework for assessing OER implementation strategies

and

A continuum of openness

Assessing the potential for openness

Stagg (2014) is also cited by Judith and Bull (2016)

While this literature has been significant in driving forward the open agenda, there has been relatively little published about the practicalities of implementing openly licensed materials in higher education courses (p. 2)

which raises the question of just how much more difficult the idea of implementing open educational practices are going to be. i.e. if sharing materials is hard enough.

OER engagement ladder

Masterman and Wild (2013) bring in the OER engaement ladder, which is talked more about in this blog post. (Interestingly the institutional repository URL for the full research report is now broken, but blog posts and slideshare resources remain)

OER engagement ladder

References

Falconer, I., Littlejohn, A., McGill, L., & Beetham, H. (2016). Motives and tensions in the release of Open Educational Resources: the JISC UKOER programme. Australasian Journal of Educational Technology, 32(4), 92–105. doi:10.14742/ajet.2258

Judith, K., & Bull, D. (2016). Assessing the potential for openness: A framework for examining course-level OER implementation in higher education. Education Policy Analysis Archives, 24(42). doi:10.14507/epaa.24.1931

Masterman, L., & Wild, J. (2013). Reflections on the evolving landscape of OER use. Paper presented at OER13: creating a virtuous circle, Nottingham, UK

Stagg, A. (2014). OER adoption: a continuum for practice. Universities and Knowledge Society Journal, 11(3), 151 – 164. doi:10.7238/rusc.v11i3.2102