Improving the happiness index
POSTED IN twitter | TAGS : conversational psychogeography, oxford, oxforduk, twitter 25 July 2012
After my talk at Oxford Geek Night I was happy to have a couple of suggestions to see if the algorithm could produce better results. One was to remove retweets from the search, which makes sense as we all know from many Twitter bios “a RT does not imply endorsement”—and that was easy to implement as the basic Twitter search api returns retweets ‘old-style’ with “RT” at the head.
The other was more complex, so I’m going to quote Owen who emailed me directly:
“This morning I thought up an analogy. Suppose you have weather readings for the last 100 days. For each day you have temperature (T), humidity (H) and mm of precipitation (P). What you’re doing is multiplying these all together, presumably because you want to get one number out. Unfortunately this number is meaningless. If you wanted to combine these quantities in some way you should really be thinking about what meaning you’re attaching to the number you get out. I’m ignoring here the fact that you multiplied them all together, when in all likelihood adding them would make more sense. I suggest it would be more meaningful to keep track of them separately, and plot three graphs instead of one. Indeed, this is what is done with weather data.
You spoke about wanting to get a measure of how much spread a set of data has. What you want is the variance, or something like it. The average (more properly called the mean) of a set of numbers is obtained by adding them all up and dividing by the total number. This tells you something very useful, but it loses all information about how spread out the information was. The variance captures that. It’s a bit tricky to calculate. I’ll try to explain it here, but you can always google for more details. Suppose you have numbers a1 up to a100. The average is M = (a1 + a2 + … + a100) / 100. The calculate the variance we have to calculate some intermediate numbers. First, you have to calculate the average. Then you have to calculate the average of each number squared: Z = (a1^2 + … + a100^2) / 100. Now the variance is V = Z – M. I know that doesn’t seem to make much sense. There is a way of calculating the variance which makes it clearer why it’s any use, but it’s a bit harder to actually implement.
You might want to square root the variance to get the standard deviation. This is measured on the same scale as the original numbers you had, so it makes a bit more sense to use that instead.”
So, @IsOxfordHappy and the location sensitive page now do both of those. I’ve removed the ‘word scale’ for the time being till I can see roughly what the numbers are. Thanks everyone for your suggestions.
Is Oxford happy?
POSTED IN art, birminghamuk, social media, twitter | TAGS : birminghamuk, conversational psychogeography, oxford, twitter 19 July 2012
After moving down to Oxford I did an update of my Birmingham Emotions conversational psychogeography project. That’s now quite simple as I have built a ‘happy monitor’ that can centre anywhere. I’m not as happy myself as I was with the results however, whether due to the increasing volume of the Tweets that it analyses or something else the rating doesn’t move around too much. Such was the problem I proposed in a very quick talk at Oxford Geek Night 27. Here are the slides from the presentation, I think the audio was being recorded and will add if I get hold of it.
I’ve already had a number of suggestions about improving the equation or analysis, if they’re code-able by me I shall try. If not I will have to ask for help…
On a side note, the whole idea of conversational psychogeography came to me when I was thinking of putting an emotional wellbeing indicator in the form of a light at the top of Birmingham’s Rotunda (see how it’s still unfinished right at the top. That was back in 2008, but it seems that London has finally installed something a little similar. Drat.
You can get twice daily Oxford updates on Twitter.
Are you in a happy place?
POSTED IN social media, social media work, twitter, web development work | TAGS : location, physchogeography 14 May 2012
On Friday I got round to doing something I’d been thinking of for a long while. I added location detection to my conversational psychogeography tool. Like the Is Brum Happy? system it takes the latest tweets around a location and rates emotionally sensitive words against a database to give scores for the happiness or emotional wellbeing of the place. If you’re using a HTML 5 browser (you probably are) you can let it reveal your location to the script (it’s not saved anywhere) and it will tell you if where you are (and a mile radius around) is happy right now.
Excellent Engagement
POSTED IN good practice, microblogging, social media, twitter | TAGS : engagement, local sports team, the simpsons, twitter 3 May 2011
Content, interaction, community—that’s what your social media profile is all about. It’s a message that seems to have hit most brands, and organisations right down to the smallest. But from what I’m seeing a lot of at the moment, there are a lot of people finding it hard to think about what to do once they get there.
There’s an episode of the Simpsons (Season Two, Episode 22), stay with me, where Mr Burns would like to be nice to Homer—but he knows nothing about him (nor really cares) so falls on the most bland of engagement:
“Hey there Mr….d’uh….Brown Shoes! How ’bout that local sports team eh?”
(Oddly for a great Simpson’s quote the video doesn’t seem to be on YouTube anywhere, but there is an audio clip here.)
Does that remind you of anything? Here’s a collection of Tweets reminding me of it that I collected on Friday:
It’s not exclusive to Twitter, nor the Royal Wedding: check out any number of Facebook fan pages or any social platform on a Friday lunchtime to see loads of “Hey guys, what are you doing this weekend. Let us know!” type-posts. They’re a close cousin of the way blogs starting up will often end their debut post with a plaintive cry of “what would you like to see?”
It is no doubt amusing to watch them all come in (and to watch the meme or cliche spread), but there’s something deeper I think—and some lessons to learn.
I think it sometimes happens because people are following what the mainstream media started to do a few years ago (‘have your say’). “Let us know!” became their coda to all stories, because they were getting to grips with the idea that people could converse and create en masse without their involvement. They were trying to channel this new thing called UCG through them so they could continue to act as gatekeepers, or perhaps they were genuinely excited by all of those pictures of snow. The TV programmes and the newspapers (and to an extent their associated online spaces) were offering an audience, much like Tony Hart in his gallery, and still do—hence the potential motivation for sharing your content through them.
Most brand social web channels don’t have such a huge audience, or if they have a big one it’s often very tightly around a subject—big wide and generic questions aren’t going to engage that audience. Your dry cleaners, or a skincare brand, aren’t the first place you think of to tell your plans for a Bank Holiday.
Possibly it also comes from a desire to “get into the conversation”, to make a brand seem like it’s one of your mates. Might work, if you’re trying to create a very small community round your social web space—if you’re usually about answering questions and sending out news, isn’t it a little odd? What are your other followers going to do with the information if you get it and and then you spread it?
Most of all, people probably do it because they see others doing the same. That’s one way to learn, but you need to think more deeply about whether any techniques apply to your situation—what they might achieve and how they might look. In essence if you’re attempting to engage around your brand then things closely related, or of direct relevance are going to hold more weight.
As a bonus here’s Mr Burn’s classic funk track ‘Look at all those idiots‘, including wailing guitar from Waylon Smithers. What’s your favourite Simpsons as metaphor for social web engagement story? Let us know!
Is Twitter about to do a mass reclaim of unused accounts?
POSTED IN twitter | TAGS : closing accounts, twitter 19 January 2011
It’s easy to sign up for a Twitter account, all you need is an email address. It used to be even easier, they weren’t even verified. I have, I estimate, about a hundred—lots used but others created for short-term projects or jokes. Some, in truth, in the same way as you register a domain name—idea half-formed but name assured.
That it was so, lead to a lot of great Twitter names claimed but unused and unloved. (@fry posted to about once very 6 months, @cat about the same)
Unless it violates a trademark, there’s no real mechanism for getting one freed up either.
Twitter has about the same sign-up to action ratio as most social web sites, but unlike Facebook for example your username, its uniqueness, its readability, matters. And those are getting used up too cheaply.
So, the first stage I think—the “where are you” email above, which I ‘ve just received. A shot that says ‘we did warn you’, when six months later—if you don’t log in— the account is closed and the name freed
How do I tell them that directing Twitpantos is a very, erm, seasonal activity?
If you’ve an account that you value, I’d take time to post every so often.
Is Twitter about to do a mass reclaim of unused accounts?
Sentiment Analysis of a Football Match
POSTED IN my projects, portfolio, social media, social media work, twitter | TAGS : football, sentiment analysis, twitter 2 December 2010
(click through for big)
Last night I turned my sentiment analysis tool on two hashtags: #bcfc and #avfc, the most widely used tags to refer to Birmingham City and Aston Villa during their League Cup quarter final game. It was a chance to see if visualising to ‘competing’ tags around the same event would be a useful exercise.
Caveats that would apply to this:
- Some people use the tags instead of team names, meaning that they might be used by people supporting the other team (or no team at all)—most fans, though seem to tag with just the hashtag representing their team.
- Some tweeters use both—these tweets could be removed technically, but make no difference to the comparative scores.
- If there’s a subject that uses more slang or metaphor than football, it’s not often discussed on Twitter.
There was a generally a downward trend throughout the match, tension? Bad football? It could have been both. The first two goals seemed to have a much bigger impact than the third—this I don’t quite understand, but it seems to be more about the tweets themselves than the tool.
I could see how a special subject-set of emotion words could be created for football, which could cope with more nuanced or unusual words. It’s something to consider.
The sentiment scores in a Google spreadsheet, csv files: #avfc tweets (657 of which were during the game), #bcfc tweets (370 during).
The obligatory Wordle:
Is Birmingham Happy?
POSTED IN future web, geodata, my projects, portfolio, social media, social media work, twitter, web development work | TAGS : anew, birminghamuk, conversational psychogeography, emotion, is brum happy, psychogeography, sentiment analysis, twitter, wormal 29 September 2010
I’ve been running a, very rough, scrape of the Birmingham (UK) based interweb for ‘emotional wellbeing’ since April of 2008. Simply put a script running twice a day read in Tweets, news headlines and (originally) blog posts and compared the words within them to a table I’d drawn up of ‘emotion’ words and fairly arbitrary scores.
It was surprisingly interesting to watch: despite its roughness, the internal consistency let patterns emerge. It broadly followed weather and sports results, with some peaks and dips you could map to specific happenings, or news stories.
It lead to a spin off focussing on Tweets from MPs, which I think influenced some of the developments that Tweetminster produced in the next year or so.
It was the patterns that lead me to keep putting off improving the algorithm, but recent Twitter API developments meant I had to do some work anyway and that (together with another project, of which more soon) gave me the impetus to give the project an overhaul. And here’s how it works now…
Twitter’s geolocation services are now much improved, so I can specify a point (the centre of Victoria Square in Birmingham) and a radius (10 miles) and get a reasonably accurate dump of Tweet data back—the algorithm calls for the most recent 1000.
Twitter is now the sole focus of data, in keeping with the ‘conversational pychogeography‘ aims of the project (in essence, words used without too much pre-meditation are more interesting than those written purely for publication). It also provides much more and more reactive data.
The words contained within these tweets are then compared to data from the University of Florida (The Affective Norms for English Words - PDF link). Within that data set each word covered (there are around a thousand in the set I’ve using) is given a score for Valence (sad to happy on a scale 0-10), Arousal (asleep to awake on a scale of 0-10) and Dominance (feeling lack of control to feeling in control on a scale of 0-10). The scores are then collated and a mean calculated. The overall emotional wellbeing score here is calculated as a mean of the three individual means, although the scores are revealed individually on the site.
I’m unsure if combining the results in this way is the best, which is why the site reveals the working — the Twitter feed just goes with one value for ease of understanding and adds a rating adjective too:
if ($brumemotion<100){$rating="fantastic";}if ($brumemotion<90){$rating="superb";}if ($brumemotion<80){$rating="good";}if ($brumemotion<70){$rating="okay";}if ($brumemotion<60){$rating="average";}if ($brumemotion<50){$rating="quiet";}if ($brumemotion<40){$rating="subdued";}if ($brumemotion<30){$rating="low";}if ($brumemotion<20){$rating="dreadful";}if ($brumemotion<10){$rating="awful";}
The Twitter feed produces results twice a day, and these scores are being saved to visualise more graphically, but the website updates every ten seconds (and will self-refresh if you stay on the site) and also displays a word cloud of the currently found ‘emotion words’:
Thoughts on further development
I’ve been experimenting with more local results (here is a version running on just one Birmingham post code — B13) as well as live graphing. I also have a version that will analyse results for a hashtag—something we may use in conjunction with the Civico player to produce ‘wormals’ (graphs of sentiment) during conferences.
But for now, I’m happy to let the new algorithm bed in—wondering about the amount of data and frequency that will be required to see the most detail—and to see what patterns we can spot.
Feedback welcome. Go see for yourself or follow on Twitter.
Loose Tweets Sink Fleets
POSTED IN good practice, twitter | TAGS : advice, tweets, twitter 24 September 2010
…or otherwise carefully crafted communications anyway.
If you’re newish to Twitter and attempting to communicate with people to achieve anything more than a way to update friends or follow people you like you might want to print this out and stick it close to your monitor.
This stuff doesn’t seem to be explained clearly enough by Twitter or people who are encouraging its use, based on the number of people I see trying to reach an audience and scuppering themselves a bit by making these mistakes:
Once you remember, pass it on.
Mind Your Language
POSTED IN good practice, twitter | TAGS : language 21 September 2010
We’re all aware, or should be, of the power of language. It’s one of the central ideas of 1984, that you can direct or restrict thought by what words you use for real concepts.
If that doesn’t convince you—it’s fiction, right—here’s a real-life example of how language alters perceptions. This is a list of language changes compiled by the Institute for Government after consultation with members of the Conservative Shadow Cabinet and advisers. It shows changes in words and you can perceive shifts in policy. Some mean exactly the same thing, and within the organisations where this sort of language is in use everyone knows what they mean—but they’re designed to feel different and direct thought.
|
IN |
OUT |
|
Delivery/roll-out |
Implementation |
|
Investment |
Spending |
|
Demand side |
Supply side |
|
Top-down |
Bottom-up |
|
Target |
Payment-by-results |
|
Regional |
Local |
|
State |
Society |
|
Strategy |
Business Plan |
|
Evidence based |
Principles based |
|
Partnership agreements |
Post-bureaucratic state |
|
Stakeholder |
Social Responsibility |
|
Active centre |
Departments |
I’m not about to rant against jargon (I’m not a fan, but industry shorthand is almost inevitable) nor am I about to suggest that most people do this sort of thing consciously. This is just to illustrate the power of word choice in communications.
Now read this:
B&H CEX? I worked it out, but it took a while. A while I could have spent following the guy.
So it reads like an invitation, but it’s to something ‘private’? Huh? Can I go?
I’ve worked out over a few years that “private view” means two things: if a show is big and important it means “no entry unless you’re invited”, if a show is small or not that well known it means “please come, there’ll be free wine and nibbles”. It’s developed as art-jargon for some reason, and people who know know and those that don’t don’t. But it does put off people who might otherwise have liked to have attended—and the smaller shows could do with that, in fact they want it.
So, just a reminder to think about the words you use if you’re trying to communicate—strip jargon and abbreviations where you can.
The future of publishing
POSTED IN future web, my projects, social media, twitter, writing work | TAGS : Dirty Bristow, magazine, online, paywall 24 August 2010
If there’s one thing that fills the web more than cat pictures it’s ruminations on the state, past or future of newspapers and magazines. The truth is old models are failing and no-one really knows. Rupert Murdoch is trying paywalls, which is a possibility for publications with existing audiences and strong brands, but what can a start-up publication do?
In my own small way I’m experimenting — this week sees the launch of a—yes—paper-based magazine that Danny Smith and I have been working on for the best part of six months. This is what it looks like:
Things we’ve worked out so far:
- Print is really expensive at small scale, but it’s still much easier to get people excited to work for and to sell than web content.
- Brand is all important: we’ve gone for wilfully obtuse and arty—we think that’s a sector we can sell to.
- A clean break between web and print means that you need to create lots of reasons for, and a fair amount of, ‘related but not similar’ content. Content that reaches the same audience, but isn’t seen as either a free or a second-rate version of what you’re asking payment for in print.
- A new thing needs its networks—we’ve tried to make sure that everyone that can feel ownership of the magazine finds it easy to talk about and share stuff about it with their networks.
- If you’ve got a brand, related events can make a fair bit of money—but they’re an additional risk. We’re operating at small scale, but publishers have tried this — Wrox Press when I worked for it’s web design offshoot was trying to maximise return on brand by conferences, it didn’t bring in enough money to save the company. It seems easier, however, to sell a specific happening via the social web than it does an ongoing concept.
- No-one’s going to pay to get past the paywall on a Twitter account—well only about ten people in my experience.
As well as being exhausting and a great hobby, there’s been a fair few opportunities to try out different promotional web-tricks that I’m going to use again. Issue two shouldn’t take so long.
