mysociety – Puffbox

27 Jul 2010

e-government, politics, technology

geodata, google, maps, mysociety

Constituency maps in under a minute
Opening up geographic data is beginning to bear fruit. MySociety’s Matthew Somerville has just unveiled MaPit, ‘our database and web service that maps postcodes and points to current or past administrative area information and polygons for all the United Kingdom.’ What that means in practice is, postcode lookups and boundary data are now just a URL away.

(Quick update: actually, not for all the United Kingdom as it turns out – the following method doesn’t work for N Ireland. See Matthew’s comment below.)

Here’s a quick example, as much for my own future reference as anyone else’s. Let’s say you wanted to generate a map of a given MP’s constituency – say Lynne Featherstone in Hornsey & Wood Green:
- You need to find the appropriate reference number for the constituency: either by browsing the list of all constituencies, or searching for places whose names begin with Hornsey. Note – these will produce nasty-looking data files, rather than pretty HTML lists. Hunt through the code, and you’ll find:
  “65883”: {“codes”: {“unit_id”: “25044”}, “name”: “Hornsey and Wood Green”, “country”: “E”, “type_name”: “UK Parliament constituency”, “parent_area”: null, “generation_high”: 13, “generation_low”: 13, “country_name”: “England”, “type”: “WMC”, “id”: 65883}
- The ‘id’ is the number you need – in this case, 65883. The MySociety API now lets you call the geometry of that area, in – among others – Google Earth’s KML format, using the following URL. (Don’t worry about the ‘4326’ here: it’s a reference to the coordinate system being used, and won’t change in this context.)
  http://mapit.mysociety.org/area/4326/65883.kml
- Conveniently, Google Maps lets you enter a KML file’s URL as a search query, and it will draw it on a map. Even more conveniently, if you add ‘output=embed’ as a search parameter, it strips away everything but the map itself. So here’s an embedded map of Lynne’s constituency, pulled into an <iframe>. Look at the source code, to see how easy it is.
Boundary data generated by MaPit.mysociety.org which contains Ordnance Survey data © Crown copyright and database right 2010; Royal Mail data © Royal Mail copyright and database right 2010 (Code-Point Open); National Statistics data © Crown copyright and database right 2010 (NSPD Open).

And thankfully, it bears a close resemblance to this map on Lynne’s own website, which took me considerably longer to churn out.
15 Oct 2008

company, e-government, technology

mysociety, puffbox, tomsteinberg

MySociety, outsourcing and precedents
Happy birthday to MySociety: five years old, and now talking in terms of 20-year plans. Tom Steinberg’s speech at last night’s birthday party contains much to ponder.

Our agreement on the basics is a given. You can do a tremendous amount of good with relatively little money, as long as you have good people involved. People who understand the context, who have a feel for the technology, and who have a passion for what they’re doing. That’s been the very basis of MySociety’s success, and (I hope) my own here at Puffbox.

‘So long as the cult of outsourcing everything computer related continues to dominate in Whitehall,’ Tom says, ‘little is going to change. [The UK government] fired everyone who could do those things, or employed them only via horribly expensive consultancies. It is time to start bringing them back into the corridors of power.’ Hmm… depends what you mean by ‘outsourcing’, and ‘bringing them back in’.

It’s an outrageous generalisation, but what the heck – my experience over the last five or so years has been that small computer-related projects done by small companies are generally successful, whereas large projects done by large companies aren’t. So if we’re talking outsourcing to those ‘horribly expensive consultancies’, by which I guess we’re talking six, seven or even (gasp) eight-figure budgets, hear hear.

But having operated as a consultancy myself for a little while now, I actually think the outsourcing of small jobs to small external operators is beneficial to all parties.
- The client generally deals with someone at, or certainly much closer to the coalface. No account managers, business analysts, project support officers, etc etc. All these people and processes are introduced to reduce risk; but in my experience, they actually increase the risk of not delivering.
- There’s an inherent benefit in doing lots of small jobs for lots of different people: you inevitably learn something new on every job, which then makes the next job even better. If you’re tied to one single government department, there just won’t be that many interesting jobs in any given year.
- ‘Coming in from outside’ gives you the right to be a bit more contrary, provocative, arrogant even. You can say the unsayable, if you like. And whilst it’s never a desirable state of affairs, you do have the right to say ‘no’ to the more insane propositions which might come your way.
- The rigidity of government grading and payscales is actually a disincentive for ‘doers’ to remain in a civil service job. To earn a ‘market wage’, you need to seek promotion to senior management levels… and with every upward step, you move further and further away from the coalface. More talking, less doing. Trust me, I’ve been there.
- It’s invariably cheaper, and often better quality.
Given the more commercial edge to the new MySociety website, it looks like I’m pushing at an open door there.

I’m a little intrigued by Tom’s comment about the ‘repurposing [of] generic new communications tools like blogs’. That has become the core of what Puffbox does, and I make absolutely no apology for it. It allows me to deliver powerful, intricate websites in double-quick time – giving end-users what they want, how they want it, whilst maintaining a straightforward back-end interface. It gives people cutting-edge tools to do their work, and hopefully makes them ask more difficult questions of the inevitably bespoke IT projects elsewhere in their work.

I detect a slightly pessimistic tone to Tom’s remarks, and not just on funding. ‘We’ve shifted the culture of government internet usage less than we might have hoped over the last five years,’ he concludes frankly. I’m more inclined to see the upside; it has shifted, and it is shifting – slowly.

I’ve written and spoken before about the power of precedent: and with every MySociety production, large or small, the precedents become stronger. (I hope the same can be said of Puffbox’s work too.) Yes, it’s taking time to see the ripple effect: but it’s definitely coming. Some of the projects I’m discussing with people just now are truly mouthwatering.
26 Aug 2008

Uncategorised

bbc, mysociety, parliament, theyworkforyou, video

MySociety completes crowd-sourced video markup

Congratulations (hardly for the first time, of course) to the MySociety crew: in less than two months, it looks like their community of volunteers has completed the work to timestamp the 42,019 video clips supplied to They Work For You by BBC Parliament, covering the entire 2007-8 parliamentary session. Hero status is rightly accorded to Abi Broom, responsible herself for more than 20% of the effort (!)… but it’s interesting to see a few familiar names in the list of ‘top timestampers’.

Of course, the time is ticking down to the start of the new Parliamentary Session, when the work starts all over again. Tom Steinberg tells me they get ‘only’ 3-400 new clips per day, so keeping up to date shouldn’t be too hard. Unless Abi gets sick, obviously. 🙂

Quite seriously, this is a fantastic achievement. The goodwill of a community of people, coupled with a trivially simple tagging tool, achieved something which – realistically – neither speech recognition technology, nor the IT budgets of Hansard and BBC Parliament (combined?) ever could. And it goes without saying, it’s largely down to the MySociety ‘brand’ of charitable activism: if Parliament had asked people to do this, do you think many would? (Not that there should be much anguish in Millbank about this invasion of ‘their’ territory; I bet Parliamentary people will be the ones most grateful for the service.)

About 13% of the video clips were tagged anonymously; my guess is that, like me, many of those were people who were searching for something on TheyWorkForYou, came across an as-yet untagged video clip, and decided to ‘leave a tip’. For me, the magic of the tool was the fact that it made this bit so easy. But that means 87% were tagged by people who went to the trouble of registering – much more than I would have guessed, although admittedly, 7 people were responsible for over 50% of the tagging.
5 Jul 2008

Uncategorised

award, bbc, mysociety, newstatesman, ukcatalyst

Here are your winners

It took a couple of days, but the list of winners from this week’s New Statesman awards has finally emerged. As predicted, MySociety didn’t go home empty-handed, with recognition for their FOI site, What Do They Know? And it’s good to see Patient Opinion getting recognition in the Community Activism category – their approach to patient feedback, and how it can help improve NHS services, is not going unnoticed.

But it’s a shock to see the BBC taking home the Democracy In Action trophy for a campaign around Radio 4’s You and Yours, which (frankly) really isn’t that great, and doesn’t demonstrate any obvious connection with democracy. Quite simply, that award should have gone to Unlock Democracy for their magnificent Vote Match site, based round the London mayoral election. You was robbed, guys.

Next up in awards season are the UK Catalyst Awards, which have now posted their shortlist. There’s an immediately obvious resemblance in the lists of nominees, but a few you probably won’t have seen before, which are worth a look at least.
23 Jun 2008

e-government, politics

mysociety, postcodes, powerofinformation

The power of postcodes
LibDem MP Lynne Featherstone has an idea. She tells Liberal Conspiracy the one IT project she’d like to see from government would be (if I can paraphrase) an email-bouncing facility, where you’d send an email (for example) to [email protected] (sic), and it would automatically get forwarded to the relevant coppers. She rightly notes that sites such as WriteToThem go most of the way towards this concept… and indeed, it’s surely the sort of project that’s right up MySociety‘s street (sorry).

Personally, I think Lynne has the right idea, but takes it to the wrong conclusion. As IT projects go, what she describes is relatively straightforward. The headaches would come in terms of (a) requiring the email recipients to keep it all up to date; and (b) the extra work generated. Reading and writing emails takes time. It would be much more efficient, in most cases, to encourage self-service via the web.

The bit Lynne gets 100% right is the power of the postcode. The UK has one of the planet’s more granular postcoding systems, with each of the nation’s 1.8m individual postcodes covering on average 15 houses. In IT terms, that’s a remarkably accurate piece of geocoding data – which virtually every adult in the country knows off by heart. You can stop people in the street, ask them, and they know it. That’s a truly awesome asset. (Which is why Ireland is now adopting a similar system, despite Post Office claims they don’t need it.)

But ask any statistician about postcodes, and they’ll glare at you – citing two problems.
- Postcode boundaries were originally designed for postal use, and don’t match the boundaries of other statistical or political geographies. I can vouch for this: they don’t even differentiate neatly between England and Wales. But as the introduction of Royal Mail’s Mailsort demonstrates, the postal purpose of postcodes isn’t what it once was.
- Postcodes change. True, but… Royal Mail issues a ‘postcode update‘ every six months. Their website explains that there’s only been one significant change, affecting only Cambridge, in the last 3 years – a lifetime in IT terms. Hey, it’s not as if they’re recoding the entire nation every other week.
I’ve never seen either of these problems as insurmountable. And I’d argue that the amazing potential stemming from universal awareness of postcodes outweighs the hassle factors.

Postcodes are the country’s greatest example of the Power Of Information. I believe we would unlock significant power if we enshrined postcodes as our key national geography, asking Royal Mail to bequeath them to the nation. All statistical and political geography should be aligned with postcodes, with a commitment not to change them for 10 years, perhaps coinciding with the Census cycle. I don’t care if there are marginally more meaningful statistical boundaries; a flawed system we all understand beats a perfect system nobody understands. Oh, and it’s cheaper too.

With improved accessibility to meaningful local data would come improved accountability. A single online search would reveal who is responsible for what in your local area; and would link to appropriate data showing whether or not they were meeting their responsibilities.

The data is all out there, free of charge in almost all cases – but the chaos of conflicting geographies makes it almost impossible to work with. I don’t believe that’s a defensible position. Power to the postcodes!

Update, 8 July: There’s now a Commons Early Day Motion on freeing postcodes, attracting decent levels of support from Labour MPs. See this post for details…
9 Jun 2008

e-government, politics

mysociety, parliament, video, youtube

Why Parliament doesn't like YouTube

LibDem MP Jo Swinson raised the subject of parliamentary video clips going on YouTube, during questions to the Leader of the House last week. You can see it below.

Helen Goodman’s response is enlightening: video material isn’t allowed to be hosted on a site where it can be searched or downloaded ‘to ensure that it is not re-edited or reused inappropriately for campaigning or satirical purposes’. In this day and age, it’s ridiculous…

…as is proven, of course, by the very fact that I can post the above video clip courtesy of TheyWorkForYou‘s new ‘mechanical Turk‘-style manual markup initiative, and BBC Parliament’s recordings.

It’s another MySociety project where my overwhelming feeling is disappointment: it’s sad that it has to come to this. And unfortunately, where Amazon can offer cash, MySociety can only offer warm feelings. But they seem to be making startling progress.

By the way… the list of signatories to Jo Swinson’s early day motion is interesting, with quite a bit of Northern Irish interest, and almost nothing from WebCameron’s Tories.
3 Jun 2008

e-government, politics

award, democracy, londonmayor, mysociety, newstatesman

The best we can do?

Nominations have closed for this, the tenth year of the New Statesman new media awards. So the winners of the five trophies are (theoretically) listed somewhere on this page. You might find a few gems you didn’t previously know about, but overall, I instinctively find the list a bit depressing.

Most nominees have only received a single nomination, in many cases by themselves, judging by the frequent use of the words ‘I’ and ‘we’. Most are pretty straightforward uses of off-the-shelf technology, by ‘one man band’ operations. And from a technical and/or creative perspective, most frankly aren’t great.

Maybe there’s a lesson in that. It’s now trivially easy to set up a passable website. Quality still takes time and skill, but you can get to the start line in next to no time, and with minimal up-front investment. From there, it’s really a question of the passion and commitment of the site ‘owner’, and its readership / community.

But I dare say it’ll be the bigger fish who will win. Expect at least one for MySociety. And I’d like to see recognition for the London Mayor ‘votematch‘ site, one of the few sites recently to really make me think. (The result it gave still haunts me.)
9 May 2008

e-government, politics

hazelblears, mysociety, opensource, petitions

Blears backs wider use of online petitions

Writing on Comment Is Free, Hazel Blears reckons Labour’s problem is that it has become distanced from its voters. ‘The problem is the powerlessness within the system for the majority of people,’ she writes. ‘People feel that their views disappear into a black hole, without the slightest echo.’

Hazel’s solution is ‘a healthy dose of direct democracy’: more directly elected mayors, a reinvigorated co-op movement, and online petitions. ‘Petitions, especially on-line, should be used to guide the deliberations of local councillors and ministers,’ she says. ‘Petitioners should be able to press for debates in council chambers and even parliament.’

If that inspires anyone to set up their own petitions system… don’t forget that the Downing Street petitions system, built by MySociety, is ‘open source’, meaning you can download and use it free of charge.
25 Mar 2008

e-government

api, datastandards, freeourbills, mysociety, parliament

Stop what you're doing and sign up

I’m not sure I need to waste my time explaining why you need to go to TheyWorkForYou and sign up to MySociety’s campaign to Free Our Bills – or rather, to have Parliamentary data marked up in mashup-friendly XML. Just compare ‘proper’ Hansard to TheyWorkForYou, and imagine the same process being done on all Parliamentary paperwork.

You may or may not be interested in the intricacies of XML parsing, or even in the uglier workings of the Houses of Parliament. But the fact is, TheyWorkForYou has become a living case study for what we want from e-government. It’s the best-practice example everyone quotes. And if they can persuade/force Parliament to work with them, it sets a valuable precedent for everyone else.

Quick update: Tom Steinberg has been in touch to say it’s not a petition, it’s ‘an action list, proper online campaign style’. Duly noted.

And when you’re done there… log into Facebook (come on, you remember) and join the campaign to allow clips of Parliament on YouTube. Useful in itself, but helpful to MPs who want to show their constituents what they’re up to. My thanks to Lynne Featherstone for the tipoff.
22 Mar 2008

e-government

api, census, datastandards, mysociety

Set the Census data free
One particularly difficult phase of my career was my time with National Statistics, in the aftermath of the 2001 Census. I tried, and ultimately failed, to persuade the organisation to recognise the tremendous asset they held in Census data, and to make wide public access a priority. I’m proud of some of the (relatively modest) things we managed to put out, but overall I’m disappointed at the many opportunities that were missed.

I remember my frustration at how everything was driven by very narrow ‘stakeholder consultation’, which ultimately resulted in the same old people asking for the same old things. The potential for civic engagement ranked well down the list of organisational priorities; the possibilities for data mashing didn’t even register. Despite the huge sums of money spent on countless consultancies, the end product was – ahem – somewhat underwhelming.

So when I discover that the 2011 Census outputs are the subject of the latest blog-based consultation, part of the Hansard Society‘s Digital Dialogues programme, of course I’m interested. And I think we all should be.

Two dates to bear in mind here. It’s nearly a year since the publication of the Mayo-Steinberg Power Of Information report, which called for ‘a strategy in which government … supplies innovators that are re-using government-held information with the information they need, when they need it, in a way that maximises the long-term benefits for all citizens.’ And just as importantly, we’re probably five years away from the first publication of census data.

This must be the first Census to take a truly web-first, and arguably even an API-first, approach to publication. Several reasons:
- Because it’s a one-off event, for which we have several years to prepare.
- Because if you think the world is web-first in 2008, just you wait and see what 2013 looks like.
- Because outsiders – from Experian to MySociety – will almost certainly do a better job than the Civil Service (sorry).
- Because it doesn’t actually prevent government doing the ‘old school’ thing itself, if it wants. In fact, if you think ‘API first’, it’ll probably result in the ‘old school’ outputs coming together easier and quicker too. Be your own client.
- Because to have any validity, the Census requires the goodwill and engagement of every person in the country. It’s one of the rare occasions where every resident puts something into a national kitty. Even if it’s only symbolic, this should be the prime example of the state giving something back to them in return.
This is one government consultation where the geek community (by which I mean us, sadly) should bring its influence to bear. We all know it’s the right thing to do; but they won’t do it unless there’s a sizeable, quantifiable demand. This would be a huge symbolic victory for openness and democratisation. This is our chance.