Toward more accurate measurements and a better understanding of students' transfer ability in solving physics problems online

Whitcomb, Kyle M.; Guthrie, Matthew W.; Singh, Chandralekha; Chen, Zhongzhou

Two earlier studies demonstrated that students' behavior data from a sequence of online learning modules can be analyzed to measure their ability to transfer their knowledge on solving one physics problem to a similar new one. In addition, adding an on-ramp module that develops basic skills improved students' transfer ability. In the current study, we improved the accuracy of the transfer measurement by identifying and excluding students who interacted with the learning modules differently from what was expected and examined two possible mechanisms by which the on-ramp module could improve student transfer. Based on a two by two framework of self-regulated learning, we hypothesized that students with a performance-avoidance oriented goal are more likely to consistently guess on their initial attempts, leaving a distinctive pattern in the log data and resulting in an underestimation of students' actual transfer ability. We divided the remaining student sample according to whether they passed the on-ramp module before or after accessing the instructional materials and compared their performance to a propensity score-matched sample from a previous semester. Improvement in transfer ability was found to primarily occur among students who passed the on-ramp module before learning. A possible explanation is that the on-ramp module served as an effective reminder for students who already possess the essential skills, but may be insufficient to develop those skills for other students. Our results suggest that online learning modules can be an accurate and flexible tool in assessing students' transfer ability. Further, our results demonstrate that the analysis of online learning data can produce more accurate and insightful results when taking into account details of student learning behavior and learning strategy.