Are Lectical Assessments fair?
Back in 1996, after many years of bemoaning the state of modern education, I set out to see if it was possible to create a new educational assessment technology focused on measuring and supporting optimal mental development. In addition to being accurate and richly formative (supportive of learning and development), the assessments created with this technology had to be fair to everyone, regardless of their native intelligence, race, ethnicity, sex, gender, belief system, or background. More than anything else, I wanted this assessment technology to work in a way that would not be biased toward a particular developmental pathway. Instead, it would celebrate diversity by making it possible to recognize the diverse pathways through which individual minds develop.
If you are already familiar with this blog, you know that my colleagues and I have created a range of tools and techniques for supporting what we call optimal, embodied learning—the kind of learning that leverages our built-in learning mechanisms to foster the growth of richly connected, curious, and healthy minds. We focus less on the content being learned and more on the way in which it is learned and how well it serves the development of the mind. You also know that when we use the word mind, we’re not just talking about knowledge and reasoning. We’re talking about the development of all aspects of mental functioning.
How to create a fair assessment
To create a fair developmental assessment, the first thing you need to do is build an understanding of how development progresses, both in terms of learning mechanisms and in terms of the ways in which particular ideas and skills develop over time. So, to get started, my colleagues and I set out to study learning. We started by collecting a wide range of cognitive-developmental assessments that required explanatory open-ended responses, then studied these responses, using a set of research methods called developmental maieutics, to exhaustively document the meanings embedded in them. Then, in 2002 we began conducting research with our own developmental assessments. By 2022, we had collected responses to over 50,000 assessments and interviews. We routinely documented the meanings contained in every single performance, constantly on the lookout for new meanings or new ways of expressing already documented meanings.
From the beginning, we have intentionally participated in a wide range of assessment projects, sampling a wide range of ages, educational levels, cultures, locations, and ethnic/racial groups. This has made it possible to document new meanings as well as the diverse ways in which different populations express the same meanings. All of the meanings we have identified are stored in a continuously curated developmental dictionary called the Lectical Dictionary.
Our electronic developmental scoring system (called CLAS), uses the Lectical Dictionary to determine the Lectical Level (complexity level) of assessment responses and place them on a lifespan scale called the Lectical Scale (a.k.a, Skill Scale).
The Lectical Dictionary, because it has been created by sampling a wide variety of populations, should be unbiased with respect to culture, ethnicity/race, and sex. This lack of bias would be reflected in Lectical Scores.
The following analyses, which explore possible scoring biases related to sex, first language, and ethnicity, were conducted on all of the adult assessments in Lectica’s database for which there were complete data. We considered anyone 18 years or older to be an adult. Ages ranged from 18 to 89. The statistical software used was Jamovi version 1.6. (See references below.)
Lectical Level & Sex
As of this writing, there are 5,717 scored assessments for which we have sex and educational level (grade) data. We used ANCOVA to ask if, after controlling for educational level, sex had an impact on adults’ Lectical Scores. We found no difference in mean Lectical Level for males and females. Mean Lectical Scores for adult males and females are the same—1097.
Please note that until 2016 we collected only two gender categories, male & female. Since then, we have been collecting more nuanced gender information and in a few years we may have enough data to report on other gender designations.
Lectical Level & First Language
As of this writing, there are 6,666 adult assessments for which we have first language information and educational level (grade). As you can see in the Estimated Marginal Means table below, the ESL (English as a second language) mean score appears to be one point higher than the mean score for native English speakers. Surprising! However, this apparent difference is not statistically significant, which means we cannot say for sure whether or not the mean scores are different from one another. It is also very small, far below measurement error, and would have no impact on decision-making, even in high-stakes situations like recruitment or college admissions.
Lectical Level & Ethnicity/Race
As of this writing, there are 6,549 adult assessments in our database for which we have both ethnicity/race and educational level data.
As you can see from the results presented below, once educational differences are taken into account, the differences between means for the four most common ethnicities/races in our database are very small (a maximum of 3 points) and not statistically significant. Moreover, a difference this small would very rarely affect decision-making, even in high-risk contexts like recruitment or college admissions.
Discussion
We’ve worked hard to create a scoring method for Lectical Assessments that is bias free. Our approach has been to ensure that we base this scoring method on exhaustive and inclusive research into the ways in which ideas and skills develop across the lifespan. The evidence so far strongly suggests that our efforts have been rewarded.
References
[1] The jamovi project (2021). jamovi. (Version 1.6) [Computer Software]. Retrieved from Jamovi site.
[2] R Core Team (2020). R: A Language and environment for statistical computing. (Version 4.0) [Computer software]. Retrieved from Cran R-project. (R packages retrieved from MRAN snapshot 2020–08–24).
[3] Fox, J., & Weisberg, S. (2020). car: Companion to Applied Regression. [R package]. Retrieved from Cran R-project.
[4] Lenth, R. (2020). emmeans: Estimated Marginal Means, aka Least-Squares Means. [R package]. Retrieved from Cran R-project.
