America: Are you happy?

[momopi]

(Click on URL to read full article with pictures)

http://www.bbc.co.uk/blogs/thereporters ... happy.html

America: Are you happy?
Maggie Shiels | 10:45 UK time, Friday, 23 July 2010

If you want to know what kind of mood America is in, check the Twitter feed.

That's what a group of scientists at Northwestern University and Harvard did and the conclusions make for interesting reading.

In a nutshell, the project Pulse of a Nation infers that the west coast of America is an altogether happier place than the east coast. The most consistently happy location was Hawaii, which one can perhaps understand; the most miserable were Mississippi and Alaska.

And the day most people are feeling grumpy is not Monday - sorry, Bob Geldof - but Thursday. That's probably because folk have had enough of the workplace, but there is still one day to go before the weekend.

"This isn't in any way a mature research project," Sune Lehmann of the Centre for Complex Network Research at Northeastern University warned me.

"But it's a cool project and promises all sorts of possibilities. And while it underlines prejudices people hold about the west coast and the east coast, it is impossible when you see the map not to make that interpretation. But again, we haven't anchored this data down."

On that issue of data, the scientists trawled through 300 million public tweets posted between September 2006 and August 2009. Those from outside the US or which didn't include their exact location were consigned to the dustbin.

The ones that remained were filtered into tweets with keywords that conveyed the mood of the twitterer. Sune and his colleagues used what he described to me as a psychological word-rating system: Affective Norms for English Words [859Kb PDF] (Anew).

A low-scoring word on Anew was rated as a negative while a high-scoring word got a positive rating. The 1,000-odd buzzwords included some that might be obvious including "happy", "sad", "glad", "love" and "joy" and some that are perhaps more obscure: "mildew", "elevator", "umbrella" and "pie".

Mr Lehmann says the inspiration for the project came from a map made by the New York Times during the Super Bowl to evaluate how people were feeling throughout the game and what the national conversation was about.

"It was a really neat visualisation that followed the whole game and let you learn a lot about America at the time. You could see who was rooting for which team and where they were, what commercials resonated, what the scores were and how people reacted," says Mr Lehmann.

"Then we had this idea that if we could connect that to a mood, it would be really interesting. We saw someone else do something simple and we thought: what would happen if we could scale it up?"

Again, Mr Lehmann stresses that "the results are not scientific" and "there are too many biases."

Since they only looked at tweets, Mr Lehmann says you could easily conclude that this is not really a true representation of America and its mood - because not everyone tweets.

He would guess that younger people rather than older use the service and that they are more likely to be pretty well educated with a higher income because tweeting means you need access to a computer or mobile phone.

Mr Lehmann says any further work would also need to include external factors: for example, looking at the weather because the belief is that people on the west coast are happier because of the climate. But then that mood might shift depending on the weather that day. Or if you live in San Francisco, the intensity of the fog might hold sway on your sense of well-being.

On a practical level, Mr Lehmann and his cohorts foresee applications including real-time reactions to a company's event or a product in different markets, responses to a President during a State of the Union address and the kind of thing that interests pollsters during election campaigns.

"The visualisations are amazing and I think it is absolutely fascinating to see the nation's mood vary in near-real time," said Johan Bollen of Indiana University in Bloomington, who was not involved in the work but who is one of several other researchers using Twitter as a tool to try to track the public mood.

He told the New Scientist that Twitter and similar services will spawn "sophisticated systems" for mood tracking.

There's an indication of the potential in Twitter's apparent success in predicting a film's box office success. Earlier this year, scientists Sitaram Asur and Bernardo A Huberman at HP Labs said that Twitter had a greater hit rate than the prediction markets in telling how well a movie will do in its opening weeks.

"The potential is there for this kind of work to be an incredibly powerful tool," says Mr Lehmann.

One of the stumbling blocks, he explains, is getting access to Twitter's fire-hose: those hundreds of millions of tweets circulating around the digital hemisphere.

Mr Lehmann cites Google's project Flu Trends as a parallel. In that case, the search giant was able to predict an outbreak by honing in on the search queries associated with flu and see where they were coming from. The data they collected consistently replicated that found by the Centres for Disease Control, but Google was two weeks ahead of the game.

Imagine if you could apply this kind of mood map to your boss's twitter feed before you head in to ask for a raise - or what about before you do a major pitch for a job or for venture capital funding?

The possibilities, Mr Lehmann says, are endless - if of course the data can be totally relied upon.