Randomized exams for large STEM courses spread via communities of practice

M. West, M. Silva, and G. L. Herman

in Proceedings of the 122nd American Society for Engineering Education Annual Conference and Exposition (ASEE 2015), 26.1302.1-26.1302.15, 2015.

We present a software system to generate, grade, and analyze individualized per-student-randomized exams. The objectives of this system are to: (1) scale exams efficiently to very large class sizes (approaching 1000 students), and (2) improve the integrity of the exam process. To achieve these objectives, we implemented software in Python and LaTeX that generates unique per-student multiple‐choice exam PDFs identified by unique error‐correcting codes. These computer‐generated exams are randomized from a tagged LaTeX source document that contains multiple variants of each question. The randomization includes randomized question variants, random question order (with constraints), and random answer order. The students take the exams on paper, coding their answers on paper Scantron sheets, which are then rescanned for import back into the de‐randomization Python grading system. The grading software produces both student scores and individualized student feedback, as well as summary statistics and analyses of the exam and questions. The results of implementing this new computer‐based randomized exam system include a dramatic reduction in student complaints about grading, an order‐of‐magnitude reduction in time‐to‐feedback, and improved instructor experience. We present detailed results including: (1) a comparison of multiple choice exams to the free response form previously used, focusing on question discrimination and predictive value, (2) students’ perceptions of exam fairness and exam‐taking experience, and (3) faculty perceptions and experiences. In addition to the randomized exam technology itself, we also analyze the spread of this technology from the source in Calculus 2 (Eng) during Fall 2012 to nine other large STEM courses in four departments by Fall 2014. We identify two key factors in this spread: (1) the use of Communities of Practice (CoPs) as "concentrators", and (2) the embedding of faculty in cross‐department teaching roles.

DOI: 10.18260/p.24639

Full text: WeSiHe2015a.pdf