Effect of discrete and continuous parameter variation on difficulty in automatic item generation

B. Chen, C. Zilles, M. West, and T. Bretl

in Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED 2019), 2019.

Automatic item generation enables a diverse array of questions to be generated through the use of question templates and randomlyselected parameters. Such automatic item generators are most useful if the generated item instances are of either equivalent or predictable difficulty. In this study, we analyzed student performance on over 300 item generators from four university-level STEM classes collected over a period of two years. In most cases, we find that the choice of parameters fails to significantly affect the problem difficulty. In our analysis, we found it useful to distinguish parameters that were drawn from a small number (< 10) of values from those that are drawn from a large—often continuous—range of values. We observed that values from smaller ranges were more likely to significantly impact difficulty, as sometimes they represented different configurations of the problem (e.g., upward force vs. downward force). Through manual review of the problems with significant difficulty variance, we found it was, in general, easy to understand the source of the variance once the data were presented. These results suggest that the use of automatic item generation by college faculty is warranted, because most problems don’t exhibit significant difficulty variation, and the few that do can be detected through automatic means and addressed by the faculty member.

DOI: 10.1007/978-3-030-23204-7_7

Full text: ChZiWeBr2019.pdf