From the archives: What is a holistic assessment?
Originally posted on December 11, 2009
During the first 10 years of my work developing Lectical Assessments, I spent a fair bit of time explaining how testing works. Gradually, I came to understand that the science behind our assessments was less important to clients than their practical benefits, so I shifted my focus away from academics toward practical application. Recently, a question about testing from a colleague sent me on a search through the archives of my original blog. Realizing that these articles contained important information that might be useful to at least a few of my readers, I decided to rescue some of them. This article on holistic assessment was written to debunk a common misunderstanding of measurement in system dynamics.
Many moons ago, when I was a hippy midwife, the idea of holism began to slip into the counterculture. A few years later, this much misunderstood notion was all the rage on college campuses. By the time I was in graduate school in the nineties there was an impassable division between the trendy postmodern holists and the rigidly old fashioned modernists. You may detect a slight mocking tone, and rightly so. People with good ideas on both sides made themselves look pretty silly by refusing, for example, to use any of the tools associated with the other side. One of the more tragic outcomes of this silliness was the emergence of the “holistic” assessment.
Simply put, the holistic assessment is a multidimensional assessment that is designed to take a more nuanced, textured, or rich approach to assessment. Great idea. Love it.
It’s the next part that’s silly. Having collected rich information on multiple dimensions, holistic test designers typically sum up a person’s performance with a holistic single number. Why is this silly? Because the so-called holistic score is pretty-much meaningless. Two people with exactly the same score may have very little in common. For example, let’s imagine that a holistic assessment examines the height and weight of individuals and these measurements are combined to yield a heightweight score. Two people, Janet and Justin take the test and receive a heightweight score of 10. But Janet is short and plump and Justin is tall and thin. The score of 10, it turns out, means something quite different for these two people.
“Holistic” scores of this kind are essentially meaningless. Heightweight is a nonsense construct. When we use assessments of this kind to measure student learning, we do students a great disservice.
This kind of holism is rooted in a misunderstanding of measurement in system dynamics. In system dynamics, multiple measurements are used to build rich models of systems. In these models, all of the measures are unidimensional. They each measure a single dimension and are never lumped together into “holistic” scores. Instead, individual measures are used to create models of systems—rich stories about the “wholes.”
In psychological assessment, dimensionality is a challenging issue. What constitutes a single dimension is a matter for debate. My research has focused on the development of a unidimensional measure of something called hierarchical complexity. Hierarchical complexity is measured with the Lectical Assessment System to obtain a complexity level score. The aim is to make this measurement as unidimensional as possible so it can be viewed as a pure measure of hierarchical complexity. We use the Lectical Assessment System, along with other metrics, to create a story about each performance on a Lectical Assessment. The story is holistic, not the metrics.
Related articles
