You point to a problem that is ubiquitous in survey research. In an ideal world, an item would mean exactly the same thing to every person who responds to it. But this just isn’t possible. People think and understand differently. Contexts vary.

Psychometric tools allow us to examine item functioning once we have enough data. We set the sample size for the first run of this survey at 500 because a psychometric analysis of a sample of that size, if it is diverse enough, can tell us which items are suffering from the problem you describe to such an extent that they are not useful.

We’re using this survey to examine trends, so we can get away with a fair amount of variation in item response and still obtain meaningful, if limited, results. If we wanted to use this survey as an assessment of an individual’s perspective, the standards for item functioning would be much higher, and we would need to go through several rounds of testing before we could claim to be making meaningful measurements.