Strategies for deploying unreliable AI graders in high-transparency high-stakes exams

S. Azad, B. Chen, M. Fowler, M. West, and C. Zilles

in Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED 2020), 2020.

We describe the deployment of an NLP-based automatic short answer grading system on an exam in a large-enrollment introductory college course. We characterize this deployment as both high stakes (the questions were on an mid-term exam worth 10% of students’ final grade) and high transparency (the question was graded interactively during the computer-based exam and correct solutions were shown to students that could be compared to their answer). We study two techniques designed to mitigate the potential student dissatisfaction resulting from students incorrectly not granted credit by the imperfect AI grader. We find (1) that providing multiple attempts can eliminate first-attempt false negatives at the cost of additional false positives, and (2) that students not granted credit from the algorithm cannot reliably determine if their answer was mis-scored.

DOI: 10.1007/978-3-030-52237-7_2

Full text: AzChFoWeZi2020.pdf