Predicting the difficulty of automatic item generators on exams from their difficulty on homeworks
B. Chen, M. West, and C. Zilles
in Proceedings of the Sixth Annual ACM Conference on Learning at Scale (L@S 2019), 2019.
To design good assessments, it is useful to have an estimate of the difficulty of a novel exam question before running an exam. In this paper, we study a collection of a few hundred automatic item generators (short computer programs that generate a variety of unique item instances) and show that their exam difficulty can be roughly predicted from student performance on the same generator during pre-exam practice. Specifically, we show that the rate that students correctly respond to a generator on an exam is on average within 5% of the correct rate for those students on their last practice attempt. This study is conducted with data from introductory undergraduate Computer Science and Mechanical Engineering courses.
Full text: ChWeZi2019.pdf