Twitter as Good as a Telephone Survey?

John is a Science contributing correspondent.

Pollsters aren’t trembling yet, but a study of Twitter posts finds that for some key political and economic questions, tracking the content of microblogs on the Internet is nearly as good as doing a traditional telephone survey.

A common complaint among people sucked into the “Twittersphere” is that no one reads their posts. A team led by Noah Smith, a computer scientist at Carnegie Mellon University in Pittsburgh, Pennsylvania, didn’t have time to read them either. But at least the researchers put that massive pile of verbiage to good use: With an average size of 11 words, the 1 billion tweets posted in 2008 and 2009 add up to an impressive cultural data set. The team used text-analysis software to detect tweets pertaining to various issues—such as whether President Barack Obama is doing a good job—and measure the frequency of positive or negative words ranging from “awesome” to “sucks.”

The results, which will be presented 25 May at a computer science conference in Washington, D.C., were surprisingly similar to traditional surveys. For example, the ratio of Twitter posts expressing either positive or negative sentiments about President Obama produced a “job approval rating” that closely tracked the big Gallup daily poll across 2009. The president’s approval slumped over the course of the year in both. (The correlation between the two was an impressive 79% when the Twitter data was averaged across chunks of several days.)

By tracking the frequency of positive or negative sentiments about people’s financial well-being—filtering for posts about saving and spending—the Twitter data also reproduced trends in some classic economic indicators such as consumer confidence.

However, the correlation broke down when the meaning of the posts was too subtle for the team's software. In the 2008 U.S. presidential election, for example, poll results for the popularity of Obama as a candidate correlated closely with how frequently his name was mentioned in tweets. But mentions of his rival “McCain” also correlated with Obama’s popularity, so the exact wording of the posts clearly mattered. An actual human would have to read all those tweets to tell which of the two candidates was actually preferred.

There are other issues too, says Nicole Ellison, a social scientist at Michigan State University in East Lansing. Just like traditional pollsters, social media researchers will have to address how representative Twitter users are of the general population. And unlike telephone surveys, small groups of people can wildly skew the results of Internet data, as the Web site 4chan demonstrated last year, when its contributors hacked Time magazine’s Internet-based poll of the world’s “most influential people” to put their Web site’s founder in first place.

So the pollsters can breath a sigh of relief—at least for now.

Posted in Math, Scientific Community