Don't go comparing

There’s a bit of a spat at the moment over Conservative (mis)use of crime stats to suggest a doubling or trebling of violent crime. The BBC’s Mark Easton has an excellent summary of the situation, which ultimately boils down to a change in how the numbers were put together:

Before 2002 the decision as to whether an incident was a violent crime had been taken by police. After 2002, officers were obliged to record all incidents as violent crimes if the alleged victim said that is what it was. The aim was to stop police fiddling the figures and to get a better picture of violence. The obvious consequence was to send the raw numbers shooting up.

Statisticians therefore warn that ‘figures before and after that date are not directly comparable’ – however, that doesn’t seem to have stopped either the Tories or, to some extent at least, the Labour government making precisely such comparisons.
Not that that’s any kind of justification, as Tory spokesman Chris Grayling attempted on the Today programme this morning:

I know there’s been a change; I also know that the Home Office has continued to use the same comparators. … As an opposition party, we don’t make the statistics. We can only use what the Home Office publishes.

My point isn’t the party politics of the situation: it’s the reminder of the pitfalls of playing with data. The launch of is rightly being hailed as a triumph: but it hands highly explosive material to eager amateurs. Some won’t notice the caveats; some won’t understand them; some may actively choose to ignore them. And some will say, ‘what does it matter, we’re all at it.’
If statistics are kept to a small clique of experts, then it’s fine to tinker with the calculations – safe in the knowledge that all the users are expert enough to understand and factor in the changes. But stats aren’t kept to the cliques any more, if ever they really were – and takes this to a whole new level.
The decision to offer the data was absolutely right in my view: in time, it should be an antidote to this. But it will come under fire at some point: and we need to have a defence ready.

Did we just win?

We’ve all learned to be cynical about government announcements – but I’m reading through today’s ‘Putting the Frontline First: Smarter Government’ paper, and I can’t help smiling. We certainly aren’t in a position where the PM can make a policy declaration, and it all falls into place by lunchtime; there are some vicious battles ahead. But there can’t be much doubt, surely, that the tide has now turned in favour of open data, accountability, transparency, third-party innovation, and technology which is both smarter and cheaper.
The paper’s ‘action points’ list looks like an agreement to do many, if not all the things we as a ‘gov 2.0’ community were asking for. A few highlights:

  • Establish common protocols for public services to exchange information
  • Consult on and release valuable public sector datasets – including mapping and postcode, Public Weather Service, detailed government expenditure, various transport and health datasets
  • Enable a single point of access for government held data through (to launch Jan 2010)
  • Launch a public consultation index through Directgov (although we’ve had this before)
  • During 2010: Ensure public consultations have online tools for interactive dialogue (er, WordPress I guess?)
  • During 2010: Ensure the majority of government-held data published in reusable form
  • By 2011: Publish all comparative data on and ensure that it is sufficiently consistent to enable cost comparisons to be made across services
  • For the longer term: Reduce consultancy spend by 50%, and communication and marketing spend by 25%

And there’s plenty more, deeper into the report: prototype building, ONS data into, ‘direct’ involvement for users in service design, local breakdowns of stimulus spending, a whole section on ‘Harnessing the power of comparative data’, and a pledge for ‘the majority of government-published information to be reusable, linked data by June 2011’. In fact, I’m struggling to think of anything on the wishlist which hasn’t been ticked off.
It’s important to see this in the widest possible context. This is just Whitehall accepting the reality many of us recognised long ago. This is a Labour government looking for causes around which to build its general election campaign; and of course, trying to steal something of a march on their rivals and likely successors: see Cameron’s pledges of June this year.
And it’s reliant on existing institutions, contract-holders and vested interests coming round to the new way of thinking. Like forcing Ordnance Survey to surrender their data. Escaping the restrictions of costly outsourcing arrangements. And embracing the tools and methods of the new ways of thinking, just as much as the mindset. Past performance doesn’t give much cause for optimism, perhaps: but at least there’s evidence of a desire to take the fight to people like OS.
The naming of the document is intriguing: having been known as the ‘Smarter Government’ paper for some time now, it emerges with the classically anodyne (and ultimately meaningless) title of ‘Putting the Frontline First’. Clearly, they were too nervous about presenting their vision as too technologically-driven – understandable, I suppose. But that’s precisely what it is.

Cameron pledges to free our data

David Cameron has taken the Conservatives’ promises on availability of public data a few steps further, in principle at least, in a speech at Imperial College on taking ‘broken politics’ into the ‘post-bureaucratic age’.
‘In Britain today, there are over 100,000 public bodies producing a huge amount of information,’ he said; ‘Most of this information is kept locked up by the state. And what is published is mostly released in formats that mean the information can’t be searched or used with other applications… This stands in the way of accountability.’ Now I’m still not convinced that there’s that much deliberate, conscious locking-up of data; but certainly, the formats in which that data is eventually made available often has the same end result.
OK, so we’re broadly agreed on the problem… what’s the solution, Dave?

We’re going to set this data free. In the first year of the next Conservative Government, we will find the most useful information in twenty different areas ranging from information about the NHS to information about schools and road traffic and publish it so people can use it. This information will be published proactively and regularly – and in a standardised format so that it can be ‘mashed up’ and interacted with.
What’s more, because there is no complete list that can tell us exactly what data the government collects, we will create a new ‘right to data’ so that further datasets can be requested by the public. By harnessing the wisdom of the crowd, we can find out what information individuals think will be important in holding the state to account. And to avoid bureaucrats blocking these requests, we will introduce a rule that any request will be successful unless it can be proved that it would lead to overwhelming costs or demonstrable personal privacy or national security concerns.
If we are serious about helping people exert more power over the state, we need to give them the information to do it. And as part of that process, we will review the role of the Information Commissioner to make sure that it is designed to maximise political accountability in our country.

Now don’t get me wrong here, it’s great to have Cameron’s explicit sign-up to the principle of data freedom, standardised formats, the presumed right of availability, and a 12-month timeframe. But it’s not really anything that the other major parties aren’t already talking about – and in the case of the current government, bringing in the Big Guns to actually do something about. OPSI’s data unlocking service, for example, is nearly a year old, and effectively answers the ‘wisdom of the crowd’ idea. Now it hasn’t been a huge hit… but the principle is already established.
And then there’s his unfortunate choice of public sector jobs as an example of what they might do:

Today, many central government and quango job adverts are placed in a select few newspapers. Some national, some regional. Some daily, some weekly. But all of them in a variety of different publications – meaning it’s almost impossible to find out how many vacancies there are across the public sector, what kind of salaries are being offered, how these vary from public sector body to public sector body and whether functions are being duplicated. Remember this is your money being put forward to give someone a job – and you have little way of finding out why, what for and for how much. Now imagine if they were all published online and in a standardised way. Not only could you find out about vacancies for yourself, you could cross-reference what jobs are on offer and make sure your money is being put to proper use.

Er, isn’t Mr C aware of the recently-upgraded Civil Service Jobs website – with its API, allowing individuals and commercial companies to access the data in a standardised format (XML plus a bit of RDF), and republish it freely? The Tories have talked about online job ads since December 2006; maybe it’s time they updated their spiel.
So what does today’s pledge boil down to? On one level it’s just headline-grabbing, bandwagon-jumping, government-bashing, policy-reannouncing rhetoric. But that’s not necessarily a bad thing. If all the work is going on already, but it isn’t well enough known, or isn’t proving as effective as it could/should be,  maybe we should be welcoming any headlines the subject manages to grab. And if Cameron’s Conservatives do take power at the next election, and truly believe in what was said today, it would be the easy fulfilment of a campaign promise to yank these initiatives out of their quiet beta periods and into the limelight.

Let freedom of information ring

It would appear that the plan to exempt MPs and Lords from Freedom Of Information provisions has been ditched. The Mail’s Benedict Brogan is trying to unpick what just happened:

Gordon Brown claims that Tories have pulled out of a cross-party deal to introduce the change. The suggestion from No10 is that up until yesterday the Tory and Labour Chief Whips were agreed that the Tories would vote with Labour in favour of the scheme. In effect the accusation is that David Cameron took fright when he realised what that would mean for his stand on transparency. The Tories are expressing mystification, suggesting that there was no deal. So either it’s embarrassment for Dave because Brown has revealed that the Tories were ready to back the exemption. Or it embarrassment for the PM because the Tories have forced him to back down.

Arguably, it doesn’t matter. A bad thing has been averted, and we – the citizens of the Internet – should take some credit, and pride in that. ‘Today we stopped moving in the wrong direction. Tomorrow we start moving the right way.‘ Not Obama’s inauguration address, as I initially assumed; that’s from Tom Steinberg’s blog post on the subject. 🙂
But it’s been a depressing couple of days, watching this come to a head. The potential implications, if such stories are true, aren’t pleasant to contemplate, if (like me) you believe it’s inherently a good thing for the country to know what its leaders are doing, and why. The two parties conspiring, behind the scenes, to get the measure through, undermining any claims they’ve ever made about transparency – and, while we’re at it, any claims of affinity to the Obama message:

As president, Obama will restore the American people’s trust in their government by making government more open and transparent and by giving regular Americans unprecedented new tools to keep track of government officials, who they are meeting with, who is giving them money and how they are spending taxpayer dollars.

It would have been sheer hypocrisy. As a small business owner, I have to be able to present receipts for every sum I try to claim back from the public purse (in the form of the Tax Man) as expenses incurred in the course of my work. I’m not allowed to deliver a top-level summary under either 9 or 26 headings. And quite simply then, MPs should have to do likewise – and be seen to do so.
And let’s give due credit to the Liberal Democrats here. It was Jo Swinson who tabled the (relatively poorly supported) EDM on Monday; and Nick Clegg had imposed a three-line whip on his MPs to oppose the move. Their credentials are reinforced today.
UPDATE: The story is evolving. ‘Tory HQ are desperate to claim that there was no deal or collusion between their backbenchers and Labour over the issue,’ says Sam Coates at The Times.  ‘The decision, apparently made in the 45 minutes between the mid-morning lobby briefing and the beginning of PMQs, looks shambolic at best – but the Conservatives’ ire has been fuelled by what was said (and left unsaid) at PMQs,’ says Niall Paterson at Sky News.

Govt report backs 'free' data

I wasn’t especially nice about the interim progress report on Power of Information Review, in that I didn’t see much specific progress being reported. So it came as a bit of a shock to discover, courtesy of the Open Rights Group, that there’s actually something really significant in it.
The interim report announced that:

HM Treasury and the Department for Business, Enterprise and Regulatory Reform (BERR) commissioned Cambridge University to undertake a study into the different models for charging for and defining public sector information. This is has (sic) been publicly released [External website].

I know, I know… that’s quite some call to action. You’re all simply dying to click on that link. But if you did, you’d eventually find your way to this report by three Cambridge University academics, published in late February, which concluded:

It was found that, in most cases, a marginal cost regime would be welfare improving – that is, the benefits to society of moving to a marginal cost regime outweighed the costs.
For registration based trading funds (DVLA, Companies House and the Land Registry) it likely (sic) that this change in charging policy could be made without the need for government to provide additional funds as any shortfall could be made up from the registration side of their activities.
For the other trading funds some direct assistance, beyond that already provided, would be required. In the case of the UKHO and the Met Office the sums involved would be limited (around £1m) but in the case of Ordnance Survey would be substantially larger (though the benefits in this case would be commensurably bigger).
One added benefit of adopting the marginal cost pricing scheme suggested by the analysis is that it would immediately address the competition concerns raised by the OFT as, a fortiori, outside organizations would now have access to ‘unrefined’ (‘upstream’) data on the same terms as the trading fund itself.

I’m no expert on all this stuff, but I know enough to recognise there are potentially staggering implications in this. And it’s surely useful ammunition for efforts like Free Our Bills.