Don't go comparing

There’s a bit of a spat at the moment over Conservative (mis)use of crime stats to suggest a doubling or trebling of violent crime. The BBC’s Mark Easton has an excellent summary of the situation, which ultimately boils down to a change in how the numbers were put together:

Before 2002 the decision as to whether an incident was a violent crime had been taken by police. After 2002, officers were obliged to record all incidents as violent crimes if the alleged victim said that is what it was. The aim was to stop police fiddling the figures and to get a better picture of violence. The obvious consequence was to send the raw numbers shooting up.

Statisticians therefore warn that ‘figures before and after that date are not directly comparable’ – however, that doesn’t seem to have stopped either the Tories or, to some extent at least, the Labour government making precisely such comparisons.
Not that that’s any kind of justification, as Tory spokesman Chris Grayling attempted on the Today programme this morning:

I know there’s been a change; I also know that the Home Office has continued to use the same comparators. … As an opposition party, we don’t make the statistics. We can only use what the Home Office publishes.

My point isn’t the party politics of the situation: it’s the reminder of the pitfalls of playing with data. The launch of is rightly being hailed as a triumph: but it hands highly explosive material to eager amateurs. Some won’t notice the caveats; some won’t understand them; some may actively choose to ignore them. And some will say, ‘what does it matter, we’re all at it.’
If statistics are kept to a small clique of experts, then it’s fine to tinker with the calculations – safe in the knowledge that all the users are expert enough to understand and factor in the changes. But stats aren’t kept to the cliques any more, if ever they really were – and takes this to a whole new level.
The decision to offer the data was absolutely right in my view: in time, it should be an antidote to this. But it will come under fire at some point: and we need to have a defence ready.

2 thoughts on “Don't go comparing”

  1. In my talk at News:Rewired, I tried to flag some of the dangers of the casual use of data, including the fact that some data points in published datasets might actually be errors… In the limiting case: “every cell is a fact, and if you want to use it in a story, check it”.

Comments are closed.