Go Seabiscuit! A new survey method encourages people to tell the truth--even when they know their opinion isn't popular.

Gaming the Polls

When an ABC News Poll asked 1028 Americans earlier this year which movie should win the 2003 Academy Awards, 42% named the eventual winner, The Lord of the Rings: The Return of the King, compared to only 16% who voted for the runner-up, Seabiscuit. But did all of these people really prefer Lord of the Rings to Seabiscuit? Or did they just pick the movie they thought was the most popular?

The difficulty of separating a respondent's true opinions from the influence of others has long made some social scientists skeptical of subjective surveys. Now, a psychologist has devised a new method that may reduce the effect of peer pressure. The technique could be especially useful for surveys that deal with touchy issues such as religion or sexuality.

According to Dražen Prelec of the Massachusetts Institute of Technology, the key idea is to introduce an element of competition--to turn the poll into a game. Besides asking participants to name their favorite movie, the pollster would ask them to predict the outcome of the poll. Each respondent would then receive a score that factored in the accuracy of their forecast and the "informativeness" of their response to their first question. For example, if you vote for Seabiscuit as your personal favorite--and if Seabiscuit does better in the poll than most people expect--you would be considered an unusually informative respondent and get a high score.

In a paper published online 14 October in Science, Prelec proves mathematically that, in this game, the optimal strategy is always to say what you really believe. Here's why. By revealing Seabiscuit as your true favorite, you increase the likelihood that the movie will do better in the poll than most people expected, thereby boosting the informativeness of your answer as well as your score. On the other hand, if you jump on the bandwagon and feign a love for hobbits, your score suffers: The informativeness of your answer is reduced because many others will have predicted that Lord of the Rings will top the poll.

The idea of scoring responses for correctness has been proposed before, but Prelec's "truth serum method" is the first one where the pollster does not have to know anything about the subject in advance. But it does have an Achilles' heel--the assumption that the respondents will play the game rationally. "It's a nice model, and really clever, but I don't think that people respond to surveys in a way that lets us use his model," says Bob Clemen, a decision analyst at Duke University. According to Clemen, people often shift into "survey mode" and make very quick, intuitive responses--exactly the opposite of the carefully considered, strategic responses Prelec's design expects of them.

Related Sites
Prelec's Science paper
Bob Clemen's Web site
A description of the Asch paradigm, an experiment that suggested that people change their answers to conform to the majority
Resources and links to additional sites about experimental and applied game theory
A working paper that describes other approaches to eliciting honest answers

Posted in Math, Social Sciences