Feb 4, 2018
What PISA measures. What we measure.
PISA is the international test that’s used to compare the effectiveness of education internationally.
Like the items in Lectical Assessments, PISA items involve real-world problems. PISA developers also claim, as we do here at Lectica, that their items measure how knowledge is applied. So, why do we persist in claiming that Lectical Assessments and assessments like PISA measure different things?
Part of the answer lies in questions about what’s actually being measured, and in the meaning of terms like “real world problems” and “how knowledge is applied.” I’ll illustrate with an example from, Take the test: sample questions from OECD’s PISA assessments.
One of the reading comprehension items in “Take the test” involves a short story about a woman who is trapped in her home during a flood. Early in the story, a hungry panther arrives on her porch. The woman has a gun, which she keeps at her side as long as the panther is present. At first, it seems that she will kill the panther, but in the end, she offers it a ham hock instead.
What is being measured?
There are three sources of difficulty in the story. It’s Lectical phase is 10c — the third phase of four in level 10. Also, the story is challenging to interpret because it’s written to be a bit ambiguous. I had to read it twice in order to appreciate the subtlety of the author’s message. And it is set on the water in a rural setting, so there’s lots of language that would be new to many students. How well a student will comprehend this story hinges on their level of understanding — where they are currently performing on the Lectical Scale — and how much they know about living on the water in a rural setting. Assuming they understand the content of the story, comprehension also depends on how good students are at decoding its somewhat ambiguous message.
The first question that comes up for me is whether or not this is a good story selection for the average 15-year-old. The average phase of performance for most 15-year-olds is 10a. That’s their productive level. When we prescribe learning recommendations to students performing in 10a, we choose texts that are about 1 phase (1/4 of a Lectical Level) higher than their current productive level. We refer to this as the “Goldilocks zone”, because we’ve found it to be the range in which material is just difficult enough to be challenging, but not so difficult that the risk of failure is too high. Some failure is good. Constant failure is bad.
But this story about a woman, a flood, and a panther is intended to test comprehension; it’s not a learning recommendation or resource. Here, its difficulty level raises a different issue. In this context, the question that arises for me is, “What is reading comprehension, when the text students are asked to decode presents different challenges to students living in different environments and performing in different Lectical Levels?” Clearly, this story does not present the same challenge to students performing in phase 10a as it presents to students performing in 10c. Students performing in 10a or lower are struggling to understand the basic content of the story. Students performing in 10c are grappling with the subtlety of the message. And if the student lives in a city and knows nothing about living on the water, even a student performing at 10c is disadvantaged.
Real world problems
Now, let’s consider what it means to present a real-world problem. When we at Lectica use this term, we usually mean that the problem is ill-structured (like the world), without a “correct” answer. (We don’t even talk about correctness.) The challenges we present to learners reveal the current level of their understandings — there is always room for growth. We think of development as a process of learning to make “better and better mistakes.” This is a VERY different mindset from the “right or wrong” mindset nurtured by conventional standardized tests.
What do PISA developers mean by “real world problem”? They clearly don’t mean “a problem without a correct answer.” Their scoring rubrics include correct, partially correct, and incorrect answers. It doesn’t get any more subtle than that. I think what they mean by “real world” is that their problems are contextualized; they are simply set in the real world. But this is not a fundamental change in the way PISA developers think about learning. Theirs is still a model that is primarily about the ability to get right answers.
How knowledge is applied
Let’s go back to the story about the woman and the panther. After they read the story, test-takers are asked to respond to a series of multiple choice and written response questions. In one written response question they are asked, “What does the story suggest was the woman’s reason for feeding the panther?”
The scoring rubric presents a selection of potential correct answers and a set of wrong answers. (No partially correct answers here.) It’s pretty clear that when PISA developers ask “how well” students’ knowledge is applied, they’re talking about whether or not students can provide a correct answer. That’s not surprising, given what we’ve observed so far. What’s new and troubling here is that all “correct” answers are treated as though they are equivalent. Take a look at the list of choices. Do they look equally sophisticated to you?
- She felt sorry for it.
- Because she knew what it felt like to be hungry.
- Because she’s a compassionate person.
- To help it live. (p. 77)
“She felt sorry for it.” is considered to be just as correct as “She is a compassionate person.” But my colleagues and I know that the ideas expressed in these two statements are not equivalent. The idea of feeling sorry for can be expressed by children as early as phase 08b (6- to 7-year-olds). The idea of compassion (as sympathy) does not appear until level 10b. And a reasonable understanding of the idea of being a compassionate person does not appear until 10c — even when the concept of compassion is being explicitly taught. Given that this is a test of comprehension — defined by PISA’s developers in terms of understanding and interpretation — doesn’t the student who writes, “She is a compassionate person,” deserve credit for arriving at a more sophisticated interpretation?
I’m not claiming that students can’t learn the word compassion earlier than level 10b. And I’m certainly not claiming that there is enough evidence in students’ responses to the prompt in this assessment to determine if an individual who wrote “She felt sorry for it.” meant something different from an individual who wrote, “She’s a compassionate person.” What I am arguing is that what students mean is more important than whether or not they get a right answer. A student who has constructed the notion of compassion as sympathy is expressing a more sophisticated understanding of the story than a student who can’t go further than saying the protagonist felt sorry for the panther. When we, at Lectica, talk about how well knowledge is applied, we mean, “At what level does this child appear to understand the concepts she’s working with and how they relate to one another?”
What is reading comprehension?
All of these observations lead me back to the question, “What is reading comprehension?” PISA developers define reading comprehension in terms of understanding and interpretation, and Lectical assessments measure the sophistication of students’ understanding and interpretation. It looks like our definitions are at least very similar.
We think the problem is not in the definition, but in the implementation. PISAs items measure proxies for comprehension, not comprehension itself. Getting beyond proxies requires three ingredients.
- First, we have to ask students to show us how they’re thinking. This means asking for verbal responses that include both judgments and justifications for those judgments.
- Second, the questions we ask need to be more open-ended. Life is rarely about finding right answers. It’s about finding increasingly adequate answers. We need to prepare students for that reality.
- Third, we need to engage in the careful, painstaking study of how students construct meanings over time.
This third requirement is such an ambitious undertaking that many scholars don’t believe it’s possible. But we’ve not only demonstrated that it’s possible, we’re doing it every day. We call the product of this work the Lectical™ Dictionary. It’s the first curated developmental taxonomy of meanings. You can think of it as a developmental dictionary. Aside from making it possible to create direct tests of student understanding, the Lectical Dictionary makes it easy to describe how ideas evolve over time. We can not only tell people what their scores mean, but also what they’re most likely to benefit from learning next. If you’re wondering what that means in practice, check out our demo.
