In late November 2009, I noted the launch of Code of Everand, a multi-player online game commissioned by the Department for Transport, aimed at teaching kids road safety in an innovative way. I found it ‘a bit tedious and mechanical’, but I acknowledged that I wasn’t exactly its target demographic.
But all sorts of alarm bells were ringing. A site like this couldn’t possibly have come cheap – but that was only the start of my concerns. It had apparently taken more than two years to build. It was built by a US-based agency. It had apparently ‘been on the brink of cancellation twice.’ And by the time it finally launched, none of the Transport staff who had originally commissioned it were still in post.
I signed off with the words ‘I imagine an opposition PQ is already in draft.’ – but as if to demonstrate just how influential this blog really is, no such PQ was ever forthcoming.
When the site’s first birthday came and went without fanfare, I could no longer suppress my curiosity… and drafted a FOI request via MySociety’s What Do They Know service. Whether the site was a success or a failure, cheap or costly, it was going to be an interesting response.
I got the response yesterday. So here we go.
That’s a grand total of £2,785,695 from project inception until the end of the current financial year. By any standards, that’s a staggering amount of money to spend on a single campaign website. Was it money well spent?
I also asked for data on user registrations and site traffic. This is a chart of new user registrations per month:
That’s a dramatic drop from a peak of 54,500 new registrations in March, to just 6,500 in April; and it’s basically been flat-lining since June. A total of 170,000 registered users, 91.8% of whom were signed up before the end of the last financial year – and the spending of a further £700k.
There’s a similar pattern to the numbers of monthly Absolute Unique Visitors, as measured by Google Analytics – although it’s curious to note the differing extents of the peaks.
The Google data regarding average time spent on the site is worth noting:
From an understandably low base, as the site built its following, the average has hovered around the 10 minute mark. That’s much more than most websites could boast, and stayed remarkably constant even as the monthly visitor numbers crashed – before suddenly jumping to nearly 18 minutes in November. Similarly, in terms of page views, it has typically recorded around 16 per visit, already very high in comparison with most sites – before shooting up to 28 in November.
I think it’s also worth mentioning that the project has social media presences in all the obvious places. And as I’m writing this, in January 2011, Code of Everand has 568 friends on Facebook – and 82 (let me confirm that: eighty-two) followers on Twitter. It’s even worse on the more child-friendly networks, where you might expect it to perform better with its notional target audience: Habbo (18 friends) and Bebo (10 – yes, ten).
Whatever interest it might have generated in those first few months, and however well it has engaged its hardcore following, it’s surely impossible to say the site has been a sustained success. Since April, its audience has all but disappeared. And yet believe it or not, just as I was writing this, there was an update on its Facebook account:
So it’s an outrageously expensive failure, right? Believe it or not, I’m still withholding judgment. Yes of course, it’s a huge amount of money: £16.33 per registered user. But it’s conceivable, just about, that it’s had its desired effect in terms of educating the youngsters it’s aimed at.
We’ll find out in due course. Transport confirmed in their response to me that:
A contract to evaluate the game was let to the Transport Research Laboratory (TRL) in March 2010. The evaluation team consists of TRL, the Serious Games Institute based in Coventry University, and Simon Christmas Ltd, an independent social researcher. We are currently in the planning stage and do not yet have a date for publishing the work and final report.
Given the collapse in visitor numbers, and the continued spending of public money on a project which would seem to have had its day – if it ever had it at all, they might want to get a move on.
Data on road casualties during 2010 are scheduled to be published in June 2011.
- If you fancy crunching the numbers for yourself, you can access the source data at What Do They Know. Thanks to the MySociety crew for making it all so easy.
- Note: I’ve since found info from the Serious Games Institute regarding their ‘five strand’ evaluation research, being led by Dr Adam Qureshi (contact details here). Sadly no publication date is quoted for their report.
Responses
Thanks for publishing this Simon, the numbers are quite astonishing.
As you say though, if it has had an positive impact on significant numbers of those users, then it could be justified. But only if.
Now, however, it does seem to have had its day.
Mike
I worked on a health marketing site for an NHS trust a few years ago, developing the back-end elements.
It was one of the most depressing things I’ve ever been involved in. The requirements were vague, but I built my part of the site with a view to long term running – however, the whole data architecture had been designed by the print designers and who in this case had absolutely no idea how to structure it correctly. This made searchability and bitrot two big problems in the long term. The site was then heavily marketed for a few weeks, and then forgotten about.
Costs weren’t insignificant either, and I’d be interested in an FOI request to find out what the agency involved was charging as well as to see what the overall cost to the taxpayer for the site was.
However, it did win awards. Not awards based on outputs, of course, but some other intangibles that I’ll probably never understand. So maybe everyone was happy and backs got patted?
Ouch. I heard £1million, although that was perhaps the approximated cost for 2009. Even so that shocked me. How does stuff like this get signed off?
OMFG 🙁
Well done for reporting this. If it hasn’t been influential before, I suspect this blog is about to become so, and that PQ will materialise sharpish. I guess many of us share frustrations about how it seems easy to spend money on ‘the shiny’ while dull stuff that increases security, capacity, resilience, provides better service, or even (gasp) generates revenue, has to be endlessly justified.
Sadly, “look like you’re doing something” often seems to be more important than “doing something” or “continue to do something”. Half the world runs on press releases, novelty, and attention-grabbing. Unfortunately, in a bid to stay “present” and “reputable”, I suspect a lot of govt projects don’t get the ongoing support they need as a result.
Excellent reporting though. Hopefully more of this will (slowly) start to turn the tide.
It’s not a question of whether this had a positive or negative effect. The real question is “Did it have the most positive effect that £2,7M could have provided?”. I suspect the answer to that question is a big fat NO.
Why do we need to burn so much money on gimmicks? When I was at school a couple of guys came to assembly class with puppets and taught us about road safety. I bet a more modern version of that method could be run in all state owned schools for under £2.7 million. The aim isn’t to make something people will go to repeatedly in order to hammer the information into their brain. The same effect can be achieved by a single, memorable display.
Nice persistence, Mr D. Superb job. #needaliedown
For £2.7 million they could have developed a mod for GTA IV based on provide road safety education to kids. Considering the amount of under age kids that seem to have their hands on the title it would have reached a larger audience.
And while they are at it, they could have added something in the mod to teach kids about the age ratings of games.
When is the evaluation available?
More importantly, as this is a Government funded project, was procurement followed and was any UK based development companies approached?
It would be interesting to see the original tender specification.
In my day, we learnt all we needed about road-safety from Frogger and we still had a farthing left over for an everlasting gumdrop.
Form a unique users perspective its important to remember that from about the time of the drop off there was PURDAH, this is the pre election period in which most “marketing” activity by Government Departments must stop, post election all marketing activity was canned as part of the spending freeze. Its therefore likely that the sites drop off was party due to a lack of marketing and promotion (although post purdah they could have done some free soicial stuff). Im not providing this as an excuse 2+million is a lot to spend but it shows how sustained marketing is needed to push something like this, or you may as well not bother.
I think David Edwards nails it there – this was the problem with the site I worked on (a long term project, really, with long term benefits but a short term marketing programme) and appears to be quite common with government sites.
I think there’s also a bit of the old “build it and they will come” attitude lingering in places as well.
I’m not entirely sure about that, chaps: a page on the Department’s own website says: ‘Whilst there are references to the DfT and THINK! within the website http://www.codeofeverand.co.uk – Code of Everand will not be actively promoted as a road safety game and no promotional resources have been produced for road safety professionals. In order to establish credibility, it is important that children ‘discover’ and go on to try the game for themselves and form their own view of the game play.’
To anyone blogging or retweeting this story: please take considerable care in what you quote, and how (as I’ve tried to do). I’ve seen a few comments which aren’t verifiable, may not be accurate, and could be considered inappropriate – or even libellous.
If you’re interested in this subject, I urge to read the coverage – and more importantly, the comments – on this piece by gaming website Joystiq. Whilst it’s obviously a self-selecting sample, you can see how the idea might not be a bad one… but the criticisms, offered by people who really know this territory (and I confess, I don’t) must be taken seriously.
That’s an awfully large spend.
I’m glad you provided some kind of cost/benefit analysis, as that’s all that matters in the end. In this case of course the benefit is near impossible to measure.
As Tomas says the best thing is to compare it with similar spending in the same area. £16/user would compare, I would imagine, with (say) a 2 hour Tufty the squirrel briefing, to take one example from my youth. I am taking those usage numbers with a pinch of salt, however. I can’t see a reason for them to fall so dramatically in 1 month unless they were being manipulated in some way to start with.
I once remember being told that 1 life is worth about £750K when it comes to road safety (that is, if a dangerous junction takes 2 lives and can be made safe for 1.5million, then it will be considered worthwhile). I expect it isn’t measured in those brutal terms directly, but that was apparently the figure if you divided safety spending by estimated lives saved per year.
Whether those numbers are correct, that type of conversation (an unemotional long hard look at the evidence) is the only one worth having.
The bit that annoys me the most is that it went to a US agency, when the UK is brimming with game development talent (and by definition you’d get half the cost back in tax if it was built here plus the profit would likely be spent here too – in an era where we actually print money it would seem a no-brainer). So a bit of probing on the commissioning process might be an idea. It smells slightly of rat to me.
Incidentally Tomas..
>> I bet a more modern version of that method could be run in all state owned schools for under £2.7 million.
There are 30000 state schools in Britain. 8 million children attend them. So that would be £90 per school, or 30p per pupil. Do the puppet show to groups of 50 at a time and that’s £15 a session. The 2 guys you mentioned would have to be prepared to pay their own travel, at the very least…
@Matt I’m being as diplomatic as I can on this one, and letting the figures speak for themselves. But I do agree: the most obviously ‘wrong’ part is the fact that it went across the Atlantic.
I think it is very important that kids are educated in all sorts of mediums including games, but the traditional methods with face to face contact are also important. I think what you are doing is brilliant.
Teaching kids road safety in an innovative way is great! but half of time they are on facebook. Maybe they should of spent the money on ads in facebook?
Great work, Simon. Do you happen to know how a US developer got selected for this project? For this sort of money, you could have started an entire games development firm in the UK that would be happily employing a significant number of people.