One of the things I’m wrestling with in my dissertation is how best to measure political sophistication or political expertise: roughly speaking, the degree to which a citizen is capable of making informed voting decisions. To do that, I’ve been using (in some places) an item-response theory model, borrowed from the psychometric and educational testing literature. These models assume that there’s a latent trait (or “ability“) that we can’t directly observe, but we can get at that ability by looking at how people answer a bunch of questions. Political expertise is a fairly straightforward application of this approach, but others have applied it to other political problems, most notably the “ideal-point estimation” problem (the most famous example of this is Keith Poole and Howard Rosenthal’s work on Congress using NOMINATE, but Simon Jackman and others have been doing important work in this area too), and the much of the field of structural equation models (or LISREL models) works on generalized versions of this problem.

What does this have to do with public opinion, per se? Well, I was thinking about the abortion issue, where each side wants to coopt as many “reasonable” people into its coalition as possible. The pro-life crowd uses slogans like “choose life” (surely nobody would “choose death,” as a recent correspondent suggested) while the pro-choice crowd tries to promote itself as “supporting family planning” (again, who would oppose that?).

Now, let’s transform this into a public opinion example. If we ask people whether or not they agree with just one of these statements, we’ll get very uninformative results; a few “die-hard” pro-choicers will recognize “choose life” as a pro-life codephrase and disagree, while a few “hardcore” pro-lifers would see “family planning” as a euphemism for abortion rights and oppose that. But most people (including myself) would agree with either statement, which would make them appear to be simultaneously “pro-choice” and “pro-life.” Is the public really this schizophrenic? Probably not; the question wording creates bias (systematic error).

Now, assume we solicit agreement or disagreement with both statements. That gives us two data points, which reflect—to some degree—the underlying attitude of the voter on abortion rights (our “latent trait” of interest). Every additional question (so long as it’s reflecting the same attitude!) gives us more leverage on the latent trait; we might, for example, ask voters whether or not they agree with abortion under particular sets of circumstances (e.g. if the mother’s life is in danger; if the child will have a birth defect; if the conception was due to rape; if the conception was incestuous; if the conception was due to the failure of other birth control; if the parent is a minor; etc.). But even using those two, very biased, questions together will give us a much better picture of the latent attitude toward abortion than just taking one of them alone. And even if the questions aren’t that biased, multiple questions beat a single question any day.

So, the moral of the story is: don’t rely on the marginals from a single survey question unless you just can’t possibly avoid it. And by “can’t possibly avoid it,” I mean “you’re stuck using data collected by people who didn’t (or couldn’t, if it was done before the 1970s) read Converse.”

If this part of public opinion is of interest to you, I strongly recommend reading The Nature and Origins of Mass Opinion, which is probably the definitive modern work in the field.