PrairieLearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning

M. West, G. L. Herman, and C. Zilles

in Proceedings of the 122nd American Society for Engineering Education Annual Conference and Exposition (ASEE 2015), 26.1238.1-26.1238.14, 2015.

We present an online problem-solving system (PrairieLearn) that is designed to facilitate learning to mastery. The objectives of this system are to: (1) enable students to practice solving randomized problem variants repeatedly until mastery, (2) incentivize students to repeat questions until mastery is achieved, and (3) provide immediate feedback about their current mastery level to the student. To achieve these objectives, we implemented an open-source web-based online system called PrairieLearn, which consists of a Node.js server and a JavaScript web-app to present randomized question variants to students. As students attempt questions, the system uses Bayesian estimation on a four-parameter item-response model to compute the real-time maximum-likelihood estimate of the student’s ability on the current topic. This estimate is shown to the student as a "mastery score", which they can increase by solving questions correctly. Because the mastery score is based on an estimate of student problem-solving ability, solving each question will result in a different change in mastery. For example, if a student has a high mastery, then successfully solving an easy question will only increase their estimated mastery by a small amount. These per-question mastery changes from answering questions are dynamically pre-calculated for each question and shown to students as “question scores”, which thus adaptively change in response to the student’s performance. Additionally, the expected value of mastery increase for each question is computed and reflected to the student as a recommendation for which question they should attempt next. Finally, the system recoreds all question attempts by students and processes this offline to learn improved models for predicting student mastery via maximum likelihood optimization.The results of using PrairieLearn over several semesters in a large engineering course (Introductory Dynamics) include: (1) significant gains in student mastery, as measured by exam results and concept inventory questions, (2) improved student satisfaction when compared to existing online problem-solving systems, and (3) high instructor satisfaction. We present data derived from students’ usage of PrairieLearn, as well as from student surveys and focus groups.

DOI: 10.18260/p.24575

Full text: WeHeZi2015.pdf