Monthly Archives: February 2011

ICTs for learning design – the first week

So, it’s the end of O-Week and a four day residential school for Graduate Diploma of Learning and Teaching Students – a post-graduate, pre-service, teaching qualification. About 100 students got a lot thrown at us for four days, now it’s time to get into some work. The following is an attempt to get some idea of and reflect upon the work I need to do with the “e-learning” course. Others will probably follow.

This course actually requires use to use a blog for reflections, so there will be more of that as well. This is a rough and ready summary of what I was doing.

Overwhelmed and Moodle

It’s the nature of this week and starting a new program that people will feel overwhelmed. Especially when there are some students who aren’t gung-ho technology users. Especially when one of the courses is around the use of Information and Communication Technologies (ICTs) in learning and teaching.

Given my background, I’m not one of those students, but I’m still feeling somewhat overwhelmed. In particular, I have this feeling that there are activities or tasks that I need to complete that I’m not aware of. I feel I’m missing something. That’s part of the reason for this post, to try and develop a list of the tasks I need to complete over the next week or so.

Beyond just the nature of starting a new program, I’m also wondering how much of this is connected with Moodle and, in particular, the design of course sites within Moodle. More on this as I generate the list.

The list

  • Add entry to “My profile” wiki (students placed in groups).
    Had some troubles with the Moodle wiki, got there in the end. Currently only two members of the group with entries. Have already experienced the problem with differentiated messaging. i.e. the other group member asked a question about the Moodle Wiki using the Moodle messaging system. I didn’t see that message until days later, long after seeing the person at the residential school.

    It is interesting that the course isn’t using some sort of aggregator to bring student blogs together, we’re being asked to add blog addresses to a wiki page.

  • Readings
    • Expert jigsaw;
      Contact one of the profile wiki members and choose a reading (about learning theories) each and use a PMI to present findings on the wiki.
    • Read the “study guide” – 3 or 4 pages
      • Do the F&S learning style test and reflect in a blog post.
      • Do a MI test and compare result with other students in a discussion forum.
      • Blog about 21st century students and Prensky.
      • Blog about some of the challenges arising from connectivism.
  • Create blog and add it to the blog list.

Blogs

The “study guide”

Most of the work for this course seems to be described in a “study guide”. A single web page within the Moodle course with some blurb, a reading/activity, and some more blurb. So here’s a summary and some reflection.

Are leaners different?

From the guide

Today’s students are fundamentally different from the learners of 40 years ago.

Mmm, not sure I agree. Sure, there is probably a greater percentage/diversity of learners at schools, especially secondary schools. But are students fundamentally different? That’s a pretty strong statement.

There is no difference

Suggestion that whether f-t-f or e-learning, that knowing the learner and their needs is a key element.

This highlights one of the misassumptions around ICTs/e-learning. People seem to think that it is different. That they haven’t been teaching and learning with technology all along. Most of the face-to-face activities we engaged with at the residential school were mediated by some form of technology. Examples included: paper handouts, sheets of paper cut up and distributed in a bag, whiteboards, rooms and furniture configured in different ways etc. I do wonder when (or perhaps if) we’ll ever get over this tendency to treat ICTs as different?

Learning styles

And so, we’re into learning styles, multiple intelligences etc as the way to know students. I have a vague disquiet about these approaches. My concerns arise from two parts. First, the tendency for these approaches to be used to pigeon hole people into a small number of categories rather than celebrate the true diversity of people. Second, the attention these approaches take away from the more complex and nuanced approaches to understanding diversity. i.e. we apply the test, identify the categories and feel satisfied that we know the students. Perhaps that’s just one source of disquiet intermixed.

Okay, so now I’m doing the survey and finding myself saying “neither a nor b”, “it depends”, “sometimes a bit of a and a bit of b”. Another concern I have about these instruments.

Seems I have to post another entry on this question.

Multiple intelligences

So, we’re meant to take this test based on Gardner’s multiple intelligences. Here are my results.

My "multiple intelligences"

As with most of these tests, it meets my basic expectations, but there continues to be a uncertainty at the back of my mind about this work. I need to read and reflect a bit more. Trying to balance the desire to dig deeper with the desire to complete the study is interesting.

The following quote from Howard Gardner is interesting in this context

I have drafted generic letters that answer the most frequent questions-for example, “Is there a test for multiple intelligences?” (Answer, “Not one that I endorse”)

The associated PDF of FAQs is interesting reading.

The following quote from those FAQs about learning styles and intelligences is interesting (to me)

Educators are prone to collapse the terms intelligence and style. For informal matters, that is no great sin. However, style and intelligence are really fundamentally different psychological constructs. Styles refer to the customary way in which an individual approaches a range of materials—for example, a playful or a planful style. Intelligence refers to the computational power of a mental system: for example, a person whose linguistic intelligence is strong is able readily to compute information that involves language. Speak of styles, speak of intelligences, but don’t conflate the two if you can help it.

And while it’s somewhat off the current topic, I need to save this comment on creativity

Finally, as stressed by my colleague Mihaly Csikszentmihalyi, creativity should not be viewed simply as a characteristic of an individual. Rather, creativity emerges from the interaction of three entities: l) the individual, with his given talents, personality, and motivation; 2) the domain—the discipline or craft in which the individual is working; 3) the field—the set of individuals and social institutions that render judgments about quality and originality

The resonance between this quote and my thoughts about creativity within education and the negative impact of institutional policies and systems – especially LMSs and how they are supported – is quite strong.

And this on how intelligences are not set in stone

The extent to which intelligences develop is a joint product of biological (genetic potential), the emphasis a culture places on an activity, the excellence of the instruction, and the motivation of the individual. Any individual can strengthen an intelligence if she is wellmotivated; if her ambient culture values that intelligence; if there are human and artifactual resources (i.e. texts, computer programs) on which she can draw.

And at the value of the MI tests

As you may know, many other people have devised MI tests. The best known is probably the MIDAS test, developed by Branton Shearer. These tests typically give a rough-and-ready sense of people’s interests and preferences. They suffer from two deficiencies: l) They don’t actually measure strengths—you would need performance tasks to determine how musically intelligent, or spatially intelligent, or interpersonally intelligent a person is; 2) The tests assume that the person has good intrapersonal intelligence—that is, he or she knows himself well. But many of us think that we know ourselves better than we really do. I doubt that anyone would score herself or himself low in the personal intelligences, but some of us must have lesser personal intelligence than others.

I have nothing against a person using the MIDAS or other measurements to learn about the idea of multiple intelligences. Indeed, people—both young and old—often find it interesting and illuminating to think about their own and/or other person’s intellectual profile. And yet, I must stress, we can only learn about a person’s intelligences if we actually measure how well they perform on tasks that presumably draw on specific intelligences.

My take away from this is that there remains significant questions about the value of using the results of a multiple intelligence test as a direct guide to a learning design. Essentially because it’s highly questionable that they actually give a valid result as to what they seek to test. A true measure, perhaps, is only possible through observing participants while performing associated tasks.

There is, however, still value in these sorts of tests as a way of getting learners to think about different intelligences within themselves and their fellow learners.

Another interesting quote is

That is, we reject the focus on a single scholastic intelligence that is measured by a certain kind of short answer test

I find it somewhat ironic that responding to “a single scholastic intelligences that is measured by…short answer test” was one of the reasons for the development of the theory of MIs, and that we are now in a situation where your MIs can be measured by a MCQ.

In summary

Briefly, my theory can reinforce the idea that individuals have many talents that can be of use to society; that a single measure (like a high stake test) is inappropriate for determining graduation, access to college, etc.; and that important materials can be taught in many ways, thereby activating a range of intelligences.

21st century learners

Ahh, Prensky, immigrants and related perspectives.

First up, the Engage me or enrage me article. The suggestion is that today’s technological environment means that all students have technologies of various types that are working hard to engage them. That’s a very broad generalisation. Then there are claims like “Many of today’s third-graders have multiple e-mail addresses” – no evidence to support this and frankly my experience doesn’t support it.

Then there’s an argument about how the top 3 selling games of the era offer vast and engaging experiences that kids aren’t getting at school – this is showing how backward school is. But this 2008 US survey suggests that the average age of gamers is 35. Only 25% of gamers is under 18, 75% are over. So it’s perhaps not the students purchasing those games.

Okay, now there is a point about students being able to handle systems or problems much more complex than what they are being given at school. There’s the example of a 4yo gamer being bored to tears by “learning games”. Mmm, final blog post on this section.

A brief overview of learning theory

So, the standard behavourism, cognitivism and constructivism summary (this seems an useful overview). Interesting that connectivism is included.

Reflections

There was a lot of reading this week, but even that only barely touched the surface of what is out there. I found it interesting, but without my prior experience this would have been a very daunting week.

Thoughts and applications of connectivism

So, post #3 for week 1 of the ICTs for Learning Design, this time on connectivism. There are two parts to the question, the second is somewhat easier and asks fo examples of how connectivism could be used in a classroom/learning context. The first is somewhat more difficult.

Position on connectivism

The question suggests that connectivism is contested by some and that some of this may arise from its challenging nature to some of the broadly accepted, established concepts or practices. The first problem with completing this arises from the fact that the guide we’re working from doesn’t really point to the folk that challenge/disagree with connectivism. This can be addressed by a quick search. That reveals a growing body of literature and challenges that can be quite complex and time-consuming to engage effectively with. Even with a history of a few years of engaging (not always deeply) with connectivism and its proponents, I don’t feel really all that comfortable enough to express a firm opinion. I’m beginning to wonder if I am over thinking this, so lets go quick and dirty for the purposes of assessment.

At the root of these learning theories/paradigms is a description of learning. I guess one approach to expressing a perspective is to examine the validity of the description of learning provided by the theories/paradigms. Based on this approach I think connectivism provides a model of learning that is perhaps closer to reality than the others. The black box of behaviourism doesn’t say anything about learning. The cognitivist approach is based on an information processing perspective. A learner as computer perspective which I don’t think captures how the brain actually works. The constructivist approach seems to wave its hands and say “learners construct knowledge”, each differently. It strikes me that there is some similarity with the network-based perspective of connectivism. i.e. students are constructing their knowledge by building and pruning networks.

That’s really broad-brush. Perhaps I should take the time to read a bit more (e.g. Kop and Hill, 2008)

One of the points in the question is

It is unsettling to be challenged about existing perceptions of “knowing”, in particular, the lack of purpose in asking our students to KNOW and be able to RECALL what they know in assessment

I’m not sure that there connectivism suggests that there would be a lack of purpose in asking students to know and recall. Connectivism, at least according to Siemens, suggests that the capacity to know (learn more) is more critical than what is known. One interpretation of this is that it doesn’t mean that what is known is unimportant, it’s just not as important as the capacity to know. In the end, demonstration of the capacity to know would seem to require an ability to demonstrate what is known.

Perhaps the view I expressed above connects more closely with Kop’s and Hill’s (2008) discussion of epistemological frameworks for learning. The final paragraph of Kop and Hill is interesting in this context

A paradigm shift, indeed, may be occurring in educational theory, and a new epistemology may be emerging, but it does not seem that connectivism’s contributions to the new paradigm warrant it being treated as a separate learning theory in and of its own right. Connectivism, however, continues to play an important role in the development and emergence of new pedagogies, where control is shifting from the tutor to an increasingly more autonomous learner.

Using this theory in the classroom

Currently I have only a theoretical/anecdotal understanding of what a high school classroom context would be like. I don’t fully appreciated the constraints of such a context, so the following will be limited by that lack of understanding.

I’ll draw on Downes comments on teaching and learning within connectivism here

to teach is to model and demonstrate, to learn is to practice and reflect

and use these within the context of teaching Information Technology – particularly programming. This is a good opportunity as it allows me to make concrete some vague ideas I’ve had for a while.

One aspect of IT is learning how to program. Often programming is taught through “pretend” authentic projects such as creating a reservation system for a restaurant. Other limitation of these projects is that the student starts from scratch and does the programming by themselves. The trouble is that increasingly most software development occurs within broader frameworks. e.g. developing a plugin for Moodle, for WordPress etc.

I’m interested in exploring how programming could be taught be encouraging students to do exactly this. Pick a module, open source application – like Moodle or WordPress – and over time develop or modify a plugin. In terms of teaching it would be my task to model and demonstrate the practices and knowledge required to do this (e.g. like my work on BIM) and the students would be required to engage in the existing developer networks around these open source projects. In terms of encouraging reflection and making connections between the students and those external, the students would be expected to maintain blogs on their practice, must like I do with BIM development.

There’s much more to this, but that’s the basics.

References

Kop, R., and A. Hill. 2008. Connectivism Learning theory of the future or vestige of the past. The International Review of Research in Open and Distance Learning 9, no. 3. http://www.irrodl.org/index.php/irrodl/article/viewArticle/523/1103%22.

Prensky, immigrants and old problems in new bottles

Okay, so blog post #2 arising from week 1 of an ICTs for Learning Design course I’m doing. This post is intended to address the question of 21st century learners. In part this connects back to a sentence from the start of this weeks study guide with which I had problems. That sentence was

Today’s students are fundamentally different from the learners of 40 years ago.

That struck me as a big claim, and to some extent it still does.

Today’s learners

In terms of today’s learners, I tend to think that they are not fundamentally different. Some (but not all) have certainly had some very significantly different experiences through the widespread availability of ICTs. I don’t, however, agree with Prensky that this experience has fundamentally changed those learners. Not the least because a significant percentage of students don’t have significant ready access to ICTs, so they can’t have changed fundamentally because of technology. Then there are questions about the type of engagement the remaining students have with technology and how much such engagement can fundamentally change them.

I currently think that today’s learners might, at a fundamental level, still have significant commonality with “yesterday’s” learners. At a less fundamental level they certainly do have some significant differences, but I am not yet convinced that those differences are fundamental. I’ll pick up on this in the next section.

Engage or enrage me

Prensky’s argument in this article goes something like this

  • Students are bombarded in their everyday life with technologies (games, phones, apps etc) that work really hard to engage them.
  • Schools just aren’t working that hard to engage them.
  • Schools need “damned good curricula gameplay” to address this problem.

I think this is an example of where there isn’t a fundamental difference between today’s and yesterday’s learners, it’s just that today’s learners have some different experiences that may well be emphasising a long-term problem. Prensky starts off his paper by dividing students into three groups:

  1. Those that are truly self-motivated.
  2. Those who go through the motions.
  3. Those who tune us out.

The last group are described as

These students are convinced that school is totally devoid of interest and totally irrelevant to their life.

I don’t see anything new in these groupings. If I go back 25+ years to my high school experiences I could quite easily place all my fellow students into those three groups. One of the major differences between then and now is that significantly more of the third group are now expected to complete years 11 and 12 of school. 25+ years ago most of the third group would have left at the end of year 10. If I go back to my parent’s days at school then the third group, such as my father, would have been leaving school much earlier.

Based on this, you might suggest the problem with “engage or enrage me” has more to do with changing societal expectations around schooling than technologies. Society now requires schools to look after the third group (and based on some recent discussion it really is a case of “babysit”) of students for much longer than prior without any fundamental change.

From another perspective the importance of learning that engages students interest and its positive effects on outcome is fairly well known/accepted. I don’t believe that the importance of engaging students has changed simply because a the most recent cohort has significantly expanded experience with technology. i.e. the change in technology is not a fundamental change.

It may be, however, a change that is important. If, and it remains a big if, students are increasingly familiar and comfortable with technologies then it might be that increase student engagement in learning may require greater and more appropriate use of engagement.

Another problem I have with Prensky’s argument is that it assumes that the fundamental change required of schools is “damned good curricula gameplay” and not a change in some of the other fundamental characteristics of schools.

Roger Shank (and many others) believe that the entire system is broken. For example, that the list of subjects which in the US owes more to the thoughts of a Harvard University professor from the 19th Century than the need to engage 21st century students.

Ira Socol goes to some lengths to show that the fundamental design of the education system (again US focused) is intended to help it fail.

So, I think there is some substance to the “engage or enrage” argument, but not in its use as the technology that is available to today’s students as being the substance. It’s the need to engage students and the difficulty of achieving such engagement, especially within the constraints of the existing education system that is important. Technology might help, but then the constraints of the existing system might prevent it from happening.

Learning styles, teaching and digital pedagogy

The following post if for the course EDED20491 and is in response to the following activity

Access the Felder and Solomon website and take the online questionnaire

  1. What is your learning style? What sorts of learning experiences would suit you best with your learning style?
  2. In a traditional classroom of 25 students, how would you support the range of learning styles each lesson?
  3. With your current knowledge of ICT, how could your design and digital pedagogy support your learners better?
  4. What sorts of profiling questions would you be asking about your learners to ensure you cater for everyone’s preferences?
  5. How does ICT support differences in learning styles?
  6. Create an entry in your blog (when it is created), and respond to these questions, and any you wish to pose, in the blog.

What is your learning style

The following image – click on it to see it in a larger form – summarises my results for the questionnaire.

Learning Styles

What that represents is that I am (supposedly)

  • A heavily reflective, rather than active learner.
    Reflective blogging, like this an related posts, is an example of a learning experience likely to suit this learning style. As suggested by Felder and Soloman
    Don’t simply read or memorize the material; stop periodically to review what you have read and to think of possible questions or applications. You might find it helpful to write short summaries of readings or class notes in your own words.

    Along similar lines, I’ve found having to teach something an effective way to learn as the act of teaching has required me to reflect upon what I know, how it is organised, how it can be explained, and what activities best encourage learning in others.

  • A heavily intuitive, rather than sensing learner.
    The explosion in information available via the Internet helps me here. I’m able to use Google and other means to find other interpretations, perspectives or theories on a topic. Establishing practices such as listening to a range of podcasts, following various edubloggers etc also open me up to alternate perspectives that enable and encourage connections. The capability to search digital information also helps in making connections to prior writings. e.g. searching my blog for posts that I believe connect in someway to a current topic. Both of which link with Felder’s and Soloman’s advice
    Ask your instructor for interpretations or theories that link the facts, or try to find the connections yourself

  • Slightly more a visual than verbal learner.
    On the verbal side the suggestion is “Write summaries or outlines of course material in your own words”. This seems to link nicely to this particular exercise. I’m essentially paraphrasing the material by Felder and Soloman and linking it to my individual context. For the visual side, the screen shot (though it’s a very text-based visual representation) of the survey results is a step in the right direction. Taking it one step further, I think the generation of slide presentations that have a highly visual component is one strategy that helps. For example, this presentation I did a couple of years ago.
  • Very slightly more global than sequential.
    Another blog post that I’m developing is essentially an attempt to develop a personal oveview of what I’m meant to be doing this week for this course. Again, a practice that links with some advice from Felder and Soloman
    Before you begin to study the first section of a chapter in a text, skim through the entire chapter to get an overview

    The post started with me skimming over the readings and activities for this week and generating a summary. When I was happy with that overview, that’s when I started working on this more detailed post in response to one of the activities.

How would you support a range of learning styles

I’m big on context. Depending on the context of the classroom of 25 students, the answers to this question would vary hugely, at least in the specifics. If I were to attempt for something abstract, the two-pronged approach would probably be

  1. Where possible adopt an appropriate array of different activities designed to support different learning styles.
    i.e. rather than simply always (or never) use classroom discussions, use them sometimes and supplement them with strategies that better suit more reflective learners. Or adopt modifications of classroom discussions that incorporate some aspect of reflection or at least time and space for reflective learners to engage.
  2. Capacity building.
    Students are not always going to find themselves in situations where their preferences are catered for. It would seem important to develop within them the capacity to deal with situations like this either through developing their skills on the other side and/or identifying strategies that help them deal with those situations.

The last point may well be essential for their school education as the context of schools may well mean that few of their learning experiences will provide exactly what their preferences require.

Design of digital pedagogy

To a large extent, I think my answer to this question is embodied in the previous two responses. First, I don’t believe there is any fundamental difference between non-digital and digital technology. Digital technology provides just another set of tools/technologies to help with learning and teaching. As it happens, I was listening to an interview of John Seely Brown this morning. In that interview Seely Brown (or it might have been the interviewer, Steve Hargadon) said something that summed up the point I’m trying to make here. The quote went something along the lines

With digital technology it’s not about the technology, but about the effects it can produce.

So, I would be using the same approach to design a digital pedagogy as I would a non-digital pedagogy. I would be aiming to fulfill the two-pronged approach from the previous question. To fulfil those two prongs I would draw on the advice and insights briefly discussed on this page and more deeply in other literature.

Which is essentially what I did in the first question where I equated the advice from Felder and Soloman with the practices/effects I was currently using this blog and other digital technologies with. i.e. I was looking for digital technologies that provide the effect suggested by the advice from Felder and Soloman.

Profiling questions

Well, I probably wouldn’t be using the Felder and Soloman learning styles questions as in the FAQ Felder explains how these questions have only been validated for college age students. The questions might retain some usefulness for younger students, however, the validity is somewhat questionable. In addition, there is the whole problem of self-reporting. i.e. I’m certain that a percentage of school students might see this as an opportunity to have some fun by answering exactly the opposite to their preference. This raises the question of whether or not there are any similar instruments that have been validated for use in school children. Not to mention validated for use with the cohort of children that I’m likely to teach.

That said, I would probably lean towards drawing on versions of these questions asked in the flow of classroom activity (rather than in a formal survey) combined with classroom observation. If subsequent research revealed validated instruments for school children, I might rely upon them.

I’m guessing that the answer to this question will have to be refined further prior to the completion of assignment 1 for the course.

How does ICT support differences in learning styles?

Again, I’d make the distinction that it is not the technology that supports the differences, but the capabilities for effects that it makes available. (This has me thinking about whether this is to fine/academic a distinction to make, for now I’ll stick with it.) Different types of technology provide effects that were previously not possible. e.g. the Blackboard provided a way for written or drawn information to be shared with a whole classroom. While a wiki allows multimedia information to be shared and modified with any group of people from anywhere with Internet access.

So, ICTs support differences in learning styles by providing effects/capabilities that fulfil some of the advice provided by Felder and Soloman. Especially when such capabilities were previously unavailable, too expensive etc. e.g. a personal blog provides a place for a reflective learner to work on and share their reflections in a way that is concrete and visible to others. Something that isn’t easily possible for a paper-based diary.

From a different perspective, ICTs support differences in learning styles because of their capability to manipulate digital information. For example, it’s possible that a verbal learner may not get much out of this blog post, but there are services that will automatically convert blog posts into audio. Hypermedia makes it easier to support both global and sequential learners in their approach to texts. Hypermedia and services like youtube make it much easier to support visual learners. e.g. 10 years ago (even 5 years ago) the readings for this course would not have been sprinkled with half a dozen videos.

From yet another perspective, ICTs support differences in learning styles to the extent that the learners, teachers, technologies, education systems and policies involved in learning allow ICTs to support differences in learning styles.

How many pages of a course profile are necessary?

This week brings the first formal tasks of my new phase as a teacher in training. We, the students enrolled in the Graduate Diploma in Learning and Teaching, are required to attend four days of a residential school. These four days coincide with Orientiation Week and we’ll be attending bits of standard O-Week, but mostly focusing on the res school.

Apart from pen and paper, the only items were advised to bring are printed versions of the course profiles. Now there was a time when the university provided print copies of the course profiles to all students. But now, mostly to save costs, these are distributed electronically and it is up to individual students to do the printing. Regardless of the course, the course profile follows a fairly standard template, which has me wondering. How many pages of a course profile do I really need to print?

After all, if it’s okay for the institution to save costs, it must be ok for me. Here’s what I found.

Course 1

3 pages of what appears to be reasonably important information, but information that is common can go. Surely it is covered in one of the myriad of guides and primers that we’ve been encouraged to read. In addition, having worked at the institution for 20 years in learning and teaching, I’m familiar with most of it.

Each assignment includes a rubric for marking. It appears to be a direct copy of what might be used by the marker. Even down to the dotted lines for comments, the lecturer’s name and a list of grades to be circled. Can’t see the need for that, especially when for 2 of the assignments that information is the only think on the page (it’s the last bit of information before the next assignment which obviously they want to start on a new page).

16 pages down to 11.

Course 2

This is the minimal course. What I think of the traditional course taught that arises when by a long-term member of staff is teaching a topic he/she knows well. Shall be interesting to see the differences evolve as the term progresses.

For example, the original course profile was only 9 pages long. The same 3 pages first removed from the profile course #1 were removed.

9 pages down to 6.

Course 3

10 pages down to 7 – same three pages.

Course 4

Again, only the same 3 pages: 13 down to 10.

Reflection

So, overall not that bad, but still 14 pages. With some work on spacing in the layout of the profile more pages could be saved. Of the 3 pages I removed, there’s an argument to remove them, but then there’s also an argument that some students have had problems by not knowing that information.

A proposed link between academic involvement and student evaluation response rate

In the following I ruminate on a possible correlation between academic participation in a course and response rates on end of term student evaluation. There are two points:

  1. It should be an relatively easy correlation to test.
  2. It would offer support for ways in which student evaluation response rates can be increased.

Understanding low response rates for student evaluation is important, at least for some institution, because they are expending significant resources in an attempt to increase response rates. Personally, I think they are generally barking up the wrong tree and that this sort of work might reveal a better tree.

Background – LMS, analytics, correlation and student participation

One aspect of the Indicators project has been using the data that is available from information systems such as an LMS to test correlations.

For example, in an early publication one of the correlations we explored was the link between student participation – measured with clicks on the course site or better yet with posts or reading of the course site discussion forum – within the LMS and student grade. Some work by Dawson (sorry don’t have the reference) had established the pattern that the greater the student LMS participation then the greater the student result. i.e. HD or A students used the LMS more.

We found this pattern with distance education students. As shown in the following graph that shows the connection between participation in a course discussion forum and student grades. (Click on the graph to see it bigger)

Average posts & replies for FLEX students

HD (high distinction – the top grade) distance education students were, on average, posting 13 replies per course discussion forum compared to 11, 9, 6, 5 and 3 for the lower grades.

But we didn’t find that on-campus, international students. As shown in the following graph using the same measure.

Average posts & replies for AIC students

The HD AIC students were actually making, on average, less replies than all other students except those that failed.

It should be remembered that we don’t know why this is happening, we’ve simply established an interesting pattern.

Another interesting pattern we established was that increase staff participation in the LMS course site also influences the correlation between student grade and student LMS participation.

Students in courses with high staff participation (staff clicked on the course site more than 3000 times in a term) gave this pattern.

Average student posts/replies on discussion forums for high staff participation courses

Students in courses with super-low staff participation (staff clicked on the course site less than 100 times in a term) gave this pattern.

Average student posts/replies on discussion forums for super low staff participation courses

Student evaluation response rates

For various reasons it has become important to some university folk to increase response rates on student evaluations. These are the anonymous, generally end-of-term, evaluation forms that ask the student about their course experience.

One institution I’m aware of is reporting figures that suggest an average response rate across all courses of about 16%. 3 of the courses has 100% response rates, but they all had 6 or less students. The highest response rate for a decent size course (n=99) was 68%.

The most visible strategy being used to increase response rates is lots of visible encouragement from the Vice-Chancellor and a 825×305 “banner” image on the LMS home page when students login.

If they want to increase response rates, I think they are barking up the wrong tree.

The right tree?

Based on my experience, observations and the above results, I believe there’s a strong correlation between the sense of responsiveness/participation students get from the academic staff teaching their courses. After all, one (but not the only one) of the reasons why I think it’s easy to get 100% response rates for a course with 6 students is that with 6 students a teacher can be very responsive.

It’s hard to get a computable indicator of that sense of responsiveness from a face-to-face classroom situation, but the above patterns from the indicators project suggest that a proxy figure – though I do admit that it is a somewhat flawed proxy – can be generate from the LMS.

I think it would be interesting to map the response rate on student evaluations with the level of staff participation in the LMS course site. If I’m right, then the best response rates should come from those courses with high staff participation.

There will be confounding situations, but in general I think the pattern would hold. In fact, the confounding situations would be interesting to investigate. What is different about those situations? What lessons might an institution learn? It might be interesting to survey some or all of these staff in order to find out what is the thinking behind their practice, is being responsive high up the list?

Of course, it’s still the wrong tree

In the end, however, there’s no getting away from the problem that such student evaluations are broken and not really worth it.

At the very least, the results of the above might just encourage a few staff to think about how they can improve student perceptions of their level of responsiveness and participation.

bim2: manage marking services – Part II

Time to continue the implementation of support services for manage marking started in the last post. Services left to be implemented are

  • Releasing marked posts.
  • Registering a blog.

Releasing marked posts

Standard order for this stuff

  1. Controller.
  2. Model.
    In this case, the model will be making changes to the database and storing the outcome.
  3. View.

Model

First, parse the parameters. There are two: marker and question. The possible combinations are

  • question and marker empty == release all marked posts for this activity.
  • question set only == release all marked posts for that question.
  • marker set only == release all marked posts for that marker.
  • marker and question = release all marked posts for that question and that marker.

So, have the model processing and updating the database. Also getting list of changes to report in the view.

The view will need to be able to translate student uid and question id into question title and user name. So need to include students and questions in the model.

View

A simple one, simply suggest success and summarise the changes made. Yep, all done. Need to do a bit more checking

Misc bugs

On manage marking there are a couple of minor bugs to fix

  • Empty table and email button showing up when there are no unregistered students.FIXED
  • The “release” link is appearing within a question heading when there are no questions to release.
    This one is probably going to be a bit more difficult to fix. Actually, the header code is correct in what it is doing. Time to check out how the rows are calculating whether there is anything marked. The display of marker details appears to be working, it’s the calculating of the stats that seems to be letting itself down.

    Ahh, all that and it’s not a code problem, but with the configuration of the users and course members. Mostly.

This is the type of bug that is going to have to be looked for in testing.

Registering a blog

This allows the coordinator to register a blog for unregistered students. It is essentially the same as the process for a student to register, just a slightly different interface. Hopefully, there can be some significant code reuse. Let’s look at how I did it for the student.

The student interface itself is a bit too complex. Registration is handled by some overly complex code within a method of one of the view. i.e. it’s not that stand alone. It is complicated because it also includes processing of the form.

Am going to have to put this into a stand along object that can be easily called elsewhere like this

$form = new process_register_form( $this->factory );
$form->process();

There are going to have to be some differences between using it with a student and with a coordinator, including

  • heading – this (including breadcrumbs) will have to change.
  • return URL – successful processing generates a “return to ??”, ?? will change based on where it’s calling from.

In terms of the heading, it’s even more complex as the student and coordinator should see radically different headings when this is called. In the end, the code looks like this

$form = new process_register_form( $this->controller );
$form->set_header( $this, "view_header", "registeration" );
$form->set_return( "?param=regOK&id=".$this->model->factory->cm->id);
$form->display();

The set_header method is used to tell $form which function to call to display the header that is required. set_return provides the URL params which let $form now where to redirect output when successfully registered.

So, time to add this in the appropriate place for manage_marking. There is some difference here in that it’s not being called from a view, but instead a controller. Another wrinkle to consider. Let’s create a view just for register and stick the call in there, this allows the view to be handled.

Ahh, no another problem, the URL for the register your blog has to be updated. Ahh, that’s the first parameter in the form. Done.

Next problem, when the student is registering their blog, the userid is based on their login, but that can’t happen when the coordinator is doing it. Has to be based on the student parameter. So, will need to pass in the userid for the student with the blog to be registered as a parameter. So this means I need to

  • Generate the form with an extra hidden parameter for coordinator. done
  • Same for student – how do I get id of browser user? (this->factory->userid) done
  • Modify processing of the form to use the hidden parameter. done

What’s next?

So, manage marking is essentially done. Time to move onto “Your students”, some new code there, also a fair bit of work to do, but should get quicker.

bim2: manage marking support services

As per the last post time to complete some support services for manage marking.

  • Releasing marked posts.
  • Displaying student details that match certain criteria.
  • Registering a blog.

Getting the controller going

First step is to set up the controller to start processing.

The current plan is that the processing will be done within the manage marking function, so tab is the same. The parameter op specifies which of the three services to execute. The view base class is already set up to allow sub-operations via the methods hash
which originally looks like this for manage marking

"marking" => array( "default" => "manage_marking" ),

which suggests only the one method. If the op URL parameter is set, then it will look for alternatives, so it changes to

"marking" => array( "default" => "manage_marking",
"view" => "view_details", 
"release" => "release_posts", 
"register" => "register_blog" ),

All that’s left is to define the methods such as view_details. Initially all with simply print statements, but subsequently with models and views.

Done.

View details

So, this should be a standard model/view combination

Model

The process here is

  • Check the additional parameters which specify which group of students to display.
  • Get the data for those students.
  • Display the details for those students.

There’s already a class that gets details for all students, seems to make sense to modify that to handle the ops.
Now have to get list of students for a marker, there is already a markers class, but it does it for all. Can it be made to work for just a single marker? Yep, that works, much the same approach as above.

So, the basic model is working. Needs a bit more work on the missing value, but will do for now.

View

The idea is to display the student details that match the chosen values. Students can be either registered or unregistered, so treat them differently.

While I wish to display it as part of manage marking, this display will also be required elsewhere. So, will implement with two views

  1. bimtwo/coordinator/view_details_view.php – responsible for the header/footer and getting the navigation to fit within manage marking.
  2. bimtwo/lib/students_view.php – to display the actual body of the page, the student data. This can be re-used elsewhere.

In fact, there is already a bimtwo_unreg_students_view class that takes care of the unregistered students.

Add in a bimtwo_reg_students_view and we’re just about there. However, it’s a little more complex as it wants to show information about the student’s feeds (how many marked, how many posted etc) and also progress against each question for the activity. This will take some time, however, it will be re-used as it serves a major component of the next tab to be implemented.

There is a bimtwo_posts class that gets all the posts for an activity, this could be modified to do the same

So, that’s basically working. It’s displaying information for both unregistered and registered students. But there’s a problem. It’s only showing details for one student and he is being listed as both registered and unregistered. Indicates a bug somewhere in the model code.

So, the model is passed to the view. It is a bimtwo_students class. This is passed to two separate views

  • unreg_students_view – which loops through an array unregistered to display details
  • reg_students-view – which loops through an array feed_details to display student details.

This is the correct behaviour. So, these are being populated incorrectly. Yep, the user is appearing in both. They shouldn’t be in the unregistered array. Ahh, unregistered is formed by this bit of code

$this->unregistered = array_diff_key( $this->user_details, $this->feed_details );

The problem is that the keys for the two different arrays (user_details and feed_details) are the ids for different objects. So, construct a hash with the right keys, use it and hey presto it’s working.

Time to put in the criteria blurb at the top, basically tells the user what criteria is being used to search for students. So, that’s working. Here’s an example, the breadcrumb navigation will need looking at.

bim2: View details

What’s next

Will post this, still have to implement

  • Releasing marked posts.
  • Registering a blog.

bim2: Coordinator’s manage marking tab

The next step in bim2 development is implementation of the manage marking tab.

What is it?

bim arises from an institutional context where it is common for courses to have at least a couple hundred students and greater than 10, sometimes greater than 20, teaching staff. The model at this institution was to have one or two of the teaching staff to take on the role of “coordinator”. The one in charge. One of the tasks for a coordinator is to manage and moderate the marking performed by other staff. Moderation is intended to ensure some consistency in marking.

To do this, the coordinator needs some way to observe how each other staff member is going with their marking and also to be able to access and view specific comments and marks awarded by the other staff. This is the purpose of the manage marking tab.

Another bim specific task was the releasing of marked posts. Since the coordinator had to moderate all marking before release, bim was designed so that only a coordinator can release marked posts. Students cannot see the mark and comments for their post until it is released.

In bim, the manage marking tab shows:

  • Unregistered students.
    This is a list of students who have not yet registered their feed with bim. bim provides a mechanism by which these unregistered students can be contacted via email using the Moodle bulk mail service.
  • Marking summary.
    A table with a row for every marker and columns for each question. In each question column it would show how many of the marker’s students have posts in the different states (i.e. submitted, marked, suspended, released, missing). Each of the different states is a link that will show the details of the students that have posts in that state. It also provides various links to release sets of posts.

What’s required?

There will four main pages for manage marking

  1. Main page.
    This shows the information described above.
  2. Results of release page.
    Showing the success (or otherwise) of an attempt to release posts.
  3. Details page.
    Shows the details of students with posts that meet certain criteria.
  4. Change blog registration.
    A way in which the coordinator can register a blog for unregistered students. This will be a page used in a number of other spots.

Each of these will have to be implemented separately, will attack them in order.

Main page

This is going to be a fairly straight-forward typical model/view combination. No form processing with this one. The only “difficult” aspect will be on deciding exactly how to create the support classes to get some of this information, rather than build it straight into the Model. Most of this data could/will be used elsewhere.

Model

The data required for this page includes

  • To calculate unregistered students
    • List of all student users for the course.
    • List of all registered feeds.
  • List of all questions for the activity.
  • List of all the markers for the course/activity and also a list of the students that are allocated to those markers.
    This is quite a complex task requiring a combination of a range of information. In bim, this was implemented as a hash where the keys were the userid for the markers and the value was another hash consisting of
    • marker – the markers id.
    • students – a hash of user details for all students allocated to the marker keyed on user id.
    • details – the markers user details.
    • statistics – a hash of stats for markers progress. i.e. how many posts are in the various states. The keys for this hash were either the question ids (which gave stats for that question) and “total” (which gave the overal total stats).

Current planning is to implement the above using the following classes

  • bimtwo_students – get all student details, including unregistered etc.
  • bimtwo_questions – this is an existing class.
  • bimtwo_markers – for the markers details.
    It has the following data members
    • DATA – hash of hashes, key is marker id, each element contains
      • firstname/lastname
      • students – hash of hashes, key is student id, value is entry groups_members
      • statistics – hash of hashes, key is question id, value is status and num in that status.
    • questions point to questions for activity

    View

    As implemented there are two main parts to the view

    1. Manage marking
      • Link to unregistered students and indication of how many there are. DONE
      • Main table for overview of marking stats for each marker. DONE
    2. Unregistered students – implement this with new view class
      • Another message about how many there are.
      • A button to email unregistered students.
      • Table listing the unregistered students and their details.

    There are also some gotchas, cases where special displays are required

    • No questions – the manage marking table doesn’t come with question columns DONE
    • No markers – no manage marking table DONE
    • No students – can’t do anything DONE

    Implementing unregistered students view

    This is a separate class as it will be used again.

    First, is there still a bulk email message service in Moodle 2? Yep, hasn’t changed. Same place. That works.

    So, now for the message and details table. Done. Here’s what it looks like

    Manage marking

    What’s left

    In terms of display, Manage marking is done. What’s left, however, are the various “services” which are linked to from manage marking. These include

    • Releasing marked posts.
    • Displaying student details that match certain criteria.
    • Registering a blog.

    I’ll leave these until the next post.

A glimmer of what I’m thinking

This time next year I should probably be teaching high school IT and maths. Because of this I’ve been thinking a lot about what type of teacher/educator/facilitator I’d like to be. What will be my ethos? I’m sure I’ll be asked that in coming weeks as I study.

This talk from Chris Lehman captures a significant part of what I’ve been thinking about.

Making high school not suck. Engaging with students in making stuff that matters.

How that can be achieved within the constraints of existing organisations should be interesting.