Diagnostic Question Clusters

Diagnostic – As the name suggests, our goal is to develop assessments that not only indicate when students are having problems learning, but why they may be having those problems.  That is, we are aiming for assessments that can inform improvements in instruction. 

Question Clusters  – The questions are designed to be used in clusters or groups, because:

  • Principled reasoning is multi-dimensional and cannot be assessed by an individual question.
  • Students’ understanding is often context specific and therefore needs to be assessed in multiple contexts or examples.

Question Formats and Interpretations – The geology question clusters are a mix of multiple choice, box-and-arrow, and drawing questions.  The biology questions are presented as multiple choice questions with stem options for essay format.  For both subject matters, foils were developed based on common wrong answers in students’ responses to open-ended versions of similar questions.  We studied students’ interpretations of the questions by having students explain in writing why they made their particular choice or through interviews with students.  We also studied students’ responses to questions in different formats.  Our work indicates that:

  • Students who make an incorrect choice have a problematic understanding of the topic being assessed.  Students who make a correct choice may still have a problematic understanding.  Thus, poor performance on the cluster of questions in multiple choice format is a reliable indicator of problematic understanding.  However, strong performance may mask problems.
  • More of students’ problematic ideas are revealed when the same questions are presented to students as multiple true/false questions where they do not know how many of the choices are correct, but must indicate whether each choice is true or false.   In this situation, in addition to indicating that the correct statement is true, some students indicate that other foils are also true.  We interpret this as meaning that students often have heterogeneous understandings that include correct ideas, but not a consistent commitment to the core principles. 
  • Using the stems of the questions as essay questions or more easily graded open-ended formats such as diagram drawing or box-and-arrow are more accurate way of assessing the fraction of students with problematic understanding and the most informative format for instructors.  See Sibley, et al (2006) & Parker, et al (in preparation).

Diagnostic Question Clusters