Matthew West

Autograding "Explain in Plain English" questions using NLP

M. Fowler, B. Chen, S. Azad, M. West, and C. Zilles

in Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE 2021), 2021.

Previous research suggests that "Explain in Plain English" (EiPE) code reading activities could play an important role in the development of novice programmers, but EiPE questions aren’t heavily used in introductory programming courses because they (traditionally) required manual grading. We present what we believe to be the first automatic grader for EiPE questions and its deployment in a large-enrollment introductory programming course. Based on a set of questions deployed on a computer-based exam, we find that our implementation has an accuracy of 87–89%, which is similar in performance to course teaching assistants trained to perform this task and compares favorably to automatic short answer grading algorithms developed for other domains. In addition, we briefly characterize the kinds of answers that the current autograder fails to score correctly and the kinds of errors made by students.

DOI: 10.1145/3408877.3432539

Full text: FoChAzWeZi2021.pdf