Correctness vs. understanding

There is a big, important difference between correctness and understanding.

Recently, I was asked by a colleague for a clear, simple example that would show how Lectical Assessment items differ from the items on conventional standardized tests. My first thought was that this would be impossible without oversimplifying. My second thought was that it might be okay to oversimplify a bit. So, here goes!

The table below lists four differences between what my colleagues and I measure with Lectical Assessments and what is measured by other standardized assessments.[1] The descriptions are simplified and lack nuance, but the distinctions are accurate.

The example

I chose a scenario-based example that we’re already using in an assessment of students’ conceptions of the conservation of matter. We borrowed the scenario from a pre-existing multiple-choice item.

The scenario

Sophia balances a pile of stainless steel wire and ordinary steel wire on a scale. After a few days, the ordinary wire in the pan on the right starts rusting.

Conventional multiple choice question

What will happen to the pan with the rusting wire?

1. The pan will move up.
2. The pan will not move.
3. The pan will move down.
4. The pan will first move up and then down.
5. The pan will first move down and then up.

(Go ahead, give it a try! Which answer would you choose?)

Lectical Assessment question

What will happen to the height of the pan with the rusting wire? Please explain your answer thoroughly.

Here are three examples of responses from 12th graders.

Lillian: The pan will move down because the rusted steel is heavier than the plain steel.

Josh: The pan will move down, because when iron rusts, oxygen atoms get attached to the iron atoms. Oxygen atoms don’t weigh very much, but they weigh a bit, so the rusted iron will “gain weight,” and the scale will to down a bit on that side.

Ariana: The pan will go down at first, but it might go back up later. When iron oxidizes, oxygen from the air combines with the iron to make iron oxide. So, the mass of the wire increases, due to the mass of the oxygen that has bonded with the iron. But iron oxide is non-adherent, so over time the rust will fall off of the wire. If the metal rusts for a long time, some of the rust will become dust and some of that dust will very likely be blown away.

Debrief

The correct answer to the multiple-choice question is, “The pan will move down.”

There is no single correct answer to the Lectical Assessment item. Instead, there are answers that reveal different levels of understanding. Most readers will immediately see that Josh’s answer reveals more understanding than Lillian’s, and that Ariana’s reveals more understanding than Josh’s.

You may also notice that Arianna’s written response would result in her selecting one of the incorrect multiple-choice answers and that Lillian and Josh are given equal credit for correctness even though their levels of understanding are not equally sophisticated.

Why is all of this important?

• It’s not fair! The multiple-choice item cheats Adriana of the chance to show off what she knows, and it treats Lillian and Josh as if their level of understanding is identical.
• The multiple-choice item provides no useful information to students or teachers! The most we can legitimately infer from a correct answer is that the student has learned that when steel rusts, it gets heavier. This correct answer is a fact. The ability to identify a fact does not tell us how it is understood.
• Without understanding, knowledge isn’t useful. Facts that are not supported with understanding are useful on Jeopardy, but less so in real life. Learning that does not increase understanding or competence is a tragic waste of students’ time.
• Despite clear evidence that correct answers on standardized tests do not measure understanding and are therefore not a good indicator of usable knowledge or competence, we continue to use scores on these tests to make decisions about who will get into which college, which teachers deserve a raise, and which schools should be closed.
• We value what we measure. As long as we continue to measure correctness, school curricula will emphasize correctness, and deeper, more useful, forms of learning will remain relatively neglected.

None of these points is particularly controversial. Most educators agree on the importance of understanding and competence. What’s been missing is the ability to measure understanding at scale and in real-time.[2] Lectical Assessments are designed to fill this gap.

[1] Many alternative assessments are designed to measure understanding — at least to some degree — but few of these are standardized or scalable.

[2] See my examination of a PISA item for an example of a typical written response item from a highly respected standardized test.

ViP info | ViP rationale