Monday, March 13, 2017

Dissertation Study Results

In my experiment, participants (N = 128) were randomly assigned to a group who saw video of me speaking in the corner (visible instructor group) or a group who saw the exact same slides and heard the exact same narration without seeing me (slides only group). I recorded the video with myself in it using Camtasia and just removed myself for the slides only version so the only difference is my visibility (see screenshots below). They completed the study in the location of their choice, not in a lab. The topic was designing PowerPoint slides using the assertion-evidence approach. I posted it on YouTube too. I would have liked to re-record it 10 more times to perfect it, but I had to move on with my life!

Visible instructor on the left - slides only on the right
During the video, questions appeared for both groups three times, at 3:06, 7:42, and 10:43 (the end), asking participants if they were paying attention, mind wandering, or doing something else entirely (multitasking). Then participants came back to the survey and…

·      Indicated their satisfaction (7-point Likert scale).
·      Responded to the 12 items of the Student-Instructor Connectedness Scale which is a validated instrument that assesses the relationship between an instructor (me) and a student (participant). Students were instructed to imagine I was really their instructor for a class. Here are two of the questions as examples: “The instructor is concerned with the needs of her students,” and “It’s easy to connect with this instructor.” Responses were on a 7-point Likert scale.
·      Indicated how many times they multitasked during the video (0-10+) and estimated how long they paid attention.
·      Provided demographic information.

Here is a very short summary of the results (the other version is 15 pages!):

First the easy one: satisfaction. Participants who saw me in the video were significantly more satisfied with the video (p = .004, r = -.25). Those numbers mean it was almost a medium level effect. Yay! This adds to some previous research that looked at preferences (not exactly satisfaction) and found conflicting info.

Overall connectedness scores were also significantly higher for the participants who saw me in the video (p = .002, r = -.29). It was just a hair short of a medium level effect. Yay again! Three of the 12 questions were not significant, but I didn’t find that surprising since they were the more personal ones that would be more difficult to get from just watching a video vs having an actual relationship with an instructor.

I am particularly excited about the connectedness finding since related studies found no difference, but they were measuring “social presence” in various ways and I used a validated instrument in a well-controlled experiment with an adequate number of participants.

Attention is the most complicated since it consisted of the questions that appeared during the video and the survey questions after. To make a long story short, there were no significant differences. If I was doing a presentation now, I’d play the sad trombone sound. I really thought I was on to something here. The overall attention levels reported are pretty consistent with other studies who found students pay attention to instructional videos about 57% of the time (kind of sad, huh?). 

I was actually surprised by all of this and expected the opposite! When I’m all done with my dissertation, I’d like to do a follow up study without the in-video questions to assess attention since they may have inadvertently helped students pay attention better (I’m not giving up on this yet) and include a test on the material.

I’m also curious what the results would have been like with a different person in the video or if the person didn’t make eye contact with the webcam (I did!). I should also note that I followed the principles of the Cognitive Theory of Multimedia Learning to develop a presentation that facilitates learning and I went through 28 slides in less than 11 minutes, meaning I really broke down the information and my slides were image-based and not too wordy. 

So, based on this information, I would recommend for instructors, particularly those teaching online, to include webcam video of themselves to improve student satisfaction and connectedness -- two aspects that can contribute to building a successful learning environment (my lit review goes into detail about this with many sources!). 

Overall, I’m glad I picked this topic because I’m genuinely interested in it and learned a lot! I have to include a big thank you to everyone who helped me with this and provided emotional support, particularly as I obsessively checked my responses in Qualtrics while the study was open (especially Avonlea and Tiffany!). I just submitted a draft of chapter 4 and will definitely be done with my PhD by June at the latest! (FYI comments are moderated due to spam but I'll check and release non-spammy ones).

1 comment:

  1. very cool, April. I think you ARE onto something, despite the results.