In a 2006 article, Wylie and Ciafolo described a technique called “single diagnostic items” that might be a great tool for teachers to gauge the impact of classroom demonstrations. Single diagnostic items focus on one important concept and “diagnose” student misconceptions about that concept. Imagine using a single item to determine what your students are learning!

Wylie and Ciafolo defined these items as “single, multiple choice questions connected to a specific content standard or objective. They have one or more answer choices that are incorrect but related to common student misconceptions regarding that standard or objective” (P. 4). 

I wanted to see how single diagnostic items worked in a real classroom, so I asked an instructor of an introductory psychology class at a local small liberal arts college for permission to work with one of her classes. We decided to focus on the topic of working memory. The text for the course did not cover this topic thoroughly and the instructor had not yet discussed this topic with the class.

My experience with single diagnostic items in the classroom.

After introducing myself and explaining the goals of the research project, I asked the class to respond in writing to the prompt: “In a few sentences, please briefly describe working memory.” Then I conducted a working memory demonstration: Students closed their eyes and mentally counted the number of windows in their house. After they finished, students closed their eyes again to “count the number of words in the sentence I just said.”

After they finished this task, students indicated whether they had to use their fingers to count when I asked them about the number of windows in their house (none of the students raised their hands). Then I asked how many used their fingers to count the number of words in the sentence (almost all the students raised their hands). Then I projected a single diagnostic item on the screen:

Why do most people use their fingers when they count the words in the sentence, but not when they count the windows?

A. Windows are visual, and visual things are easy to process.

B. Most people are visual learners.

C. The windows are in long term memory, but the words are in short term memory.

D. Familiarity – I’m more familiar with my windows than I am the words in that sentence, so that task is harder.

E. I can picture the windows but I can’t picture the words, and that has something to do with it.

F. Working memory must process words and pictures differently.

Students then indicated their response to this item (using their cell phones and the website Poll Everywhere: http://www.polleverywhere.com/). We briefly discussed the diversity of their responses, shown here:

In our discussion, the students pointed out that at least one student in the class chose each of the possible responses. We discussed the frequency of the different responses: most students chose answer C (“The windows are in long term memory, but the words are in short term memory”) or answer E (“I can picture the windows but I can’t picture the words, and that has something to do with it”). We briefly discussed the diversity of responses and concluded the data indicated the class didn't have a common explanation for why the word counting task required almost everyone to count on their fingers and the windows counting task did not.

Then I explained the origin of the task: Baddeley and Hitch (1974) established that working memory is an active system made up of separate elements that deal with different kinds of information differently. To complete the “counting the windows” task, first working memory determines that the windows need to be pictured and then counted (“central executive” function). Then working memory activates the element that handles words and numbers in order to count the windows (“phonological loop”), and the element that can picture each window visually ("visuo-spatial sketchpad”). When faced with the “count the number of words in the sentence I just said” task, the central executive encounters a problem. The phonological loop has to repeat the words in the sentence, but the visuo-spatial sketchpad can’t count, so most people have to use their fingers to complete the task.

After explaining the working memory research and terminology to the class, the students again wrote answers to the writing prompt:

In a few sentences, please briefly describe working memory.

Once again, students used their cell phones to vote on the correct answer to the diagnostic item:

The class discussed the data results and agreed the memory demonstration and explanation changed their conceptions and understandings about the nature of working memory. Almost everyone in the class accepted that answer F: “working memory must process words and pictures differently” was the most correct answer. We discussed the two previous most common answers (C and E) and the class were able to describe in what ways those responses were correct and incorrect.

Later, I analyzed the students’ written responses to look for other evidence of changes in their understanding of the working memory concept. I created a short rubric to score students’ pre and post writing responses:

Each student response was scored by my colleague and I, without knowing which responses were “pre” and which were “post.” The scoring data results (shown in the graph below) also indicated changes in student's understanding of the working memory concept.

Single diagnostic items like this one could be used to assess the effectiveness of the classroom demonstration about operational definitions. This “effectiveness data” could support decision making about the most successful demonstrations and which need to be modified. These same data could have multiple formative purposes:

- Teachers can regroup students into discussion groups based on their responses and ask groups to process the rationale behind their answers.

- Heterogeneous discussion groups might be useful, with each student discussing their different answer with the goal of the group moving toward a consensus conclusion.

- Teachers could use the two most common answers and use other classroom demonstrations/activities to focus on those misconceptions directly.

All these formative uses of the assessment data share a common characteristic: data from this one item focused specifically on student misunderstandings about an important concept. This focus on the misconceptions the students demonstrated, addressed students thinking both actively and directly. The assessment data works strategically to inform instruction by the teacher and metacognition by the students.

How to develop Single Diagnostic Items

Developing single-diagnostic items does require teachers to invest time in the item development process, but can save time in the classroom by efficiently providing valuable information about student misconceptions. One item-development process is described below:

1. Gather teachers who teach the same/similar content. Writing single-diagnostic items requires “deep” content knowledge, and is best done with a group of experienced teachers.

2. Choose a “big idea” to focus on. Single-diagnostic items take a while to write, so the group should spend its time focusing on an idea/concept/etc. that is a “big deal.” Some authors call these “hinge” or “threshold” concepts: ideas that students need to understand well in order to make progress in the discipline.

3. Ask the group to list misconceptions about the “big idea” (another way to phrase this task is to ask: “How do students go wrong about this idea?"). List all the misconceptions the group develops, then look at the list and collapse any similar ideas into appropriate categories.

4. Write a stem for the single-diagnostic item that will require students to use the “big idea.”

5. Write options for the single-diagnostic idea, one option per misconception and one possible correct answer. Note: multiple correct answers can be included, and the group should end up with one (and only one) option for each misconception. Ideally, if a student chooses an incorrect option, teachers should be confident the student did so because they are laboring under that specific misconception.

6. Test the item with real students. Participating teachers should use the item in class, and ask students who choose an incorrect response WHY they chose that response to test the relationships between incorrect options and misconceptions.

7. Revise based on feedback.

References:

Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 8, pp. 47-89). New York, NY: Academic Press

Wylie, C., & Ciofalo, J. (2006). Using diagnostic classroom assessment: One question at a time. Teachers College Record, Jan. 10, 2006, 1-6.

[Note: this blog post originally appeared on the Noba Blog].

Poster Image by Olya Kobruseva from Pexels