Friday, August 24, 2018

the danger of believing 'official statistics'...

Some of you may recall how I've argued in the past about how we should take the 'encouragements' and direction offered by our sector bodies with more than a generous handful of salt...

And I've based those arguments on where such bodies get their money from, and whom they're trying to curry favour with rather than asking us what their priorities should be; (don't believe me? Think back to the last time a sector body you're in membership of openly asked you what they should be doing on your behalf, openly published the results, and then reported on how far they'd delivered against those).

Some of you also know that I have a habit at looking at published research and data - not as a trained mathematical/economic analyst, but rather as an 'enthusiastic amateur', to see what new (or contradictory) stories they might offer to help us better understand ourselves and the issues we face.

And my latest 'rummaging' concerns the NCVO's Almanac and Social Enterprise UK's 'State of the Sector' survey. And the reason for picking these two august bodies? I'm refreshing some of the training courses and materials I deliver around governance in the sector, and though it might be interesting to see what both sources have to say about latest trends (as they both map legal forms adopted in the sector, and both published their current findings for the same year: 2017).

As with all surveys from different bodies, they've framed and categorised data relating to the same question against a range of different options and headings, but with a bit of rough transposition on my part, what seems to emerge is a picture of some confusion:



Quite some difference.

But so what you might say (and perhaps rightly so) - NCVO is concerned with those organisations who are predominately volunteer dependent (hence the title), whereas SEUK is concerned with enterprises that are trading - so we'd expect to see a difference, right?

Well, here's why I think this is more of an issue for us than we might think it is:

1) Policy makers, funders, and commissioners all group charities and social enterprises together into a single pot (hence the term 'civil society'). 
They don't differentiate in the way that our sector bodies do, so when they come to make policy and pass legislation that will affect us, how can we have confidence they're making the right choices when the data we're presenting them with to do so appears to be so contradictory?

2) Where the data comes from. 
NCVO's Almanac draws upon several external, objective, and credited sources, so should be pretty sound. SEUK's data is largely a collation of the returns they've received to a call for survey responses from groups in the sector - so their data is only as good as the people who fill out the forms. And that could be more of a problem than we'd like to admit: as a result they list sole traders as recognised legal types of social enterprise in their report (not sure anyone else would?), and 6% of all the CICs who responded weren't able to identify or be categorised as to whether they have share capital or not which, when added together with the other 'uncertain' responses, shows that over 10% of all respondents whose data has been collated and published as reflecting the realities of the sector today don't know what their legal form is - so how can we trust that they'll be any more accurate in other data that they give?

Let me put this another way - if we were building a house wouldn't we want to have confidence in the nature of the foundations, and that the way they'd been created would subsequently stand up to scrutiny from a variety of inspectors and other builders to ensure that what we're building doesn't come crashing down around us in the future when the weather gets rough..? 
If the data we use to act as the comparable 'bedrock' in shaping support, interventions, and lobbying for the sector is found to be suspect in the future, isn't there therefore a risk that a lot of efforts now would be wasted, hence why it's important to make sure we test such data today..?

As with any mapping exercise, there will always be ways in which the methodologies used, and the way the analysis on data captured can be criticised. My reason for doing so here isn't to belittle or reduce the efforts of these 2 bodies, but rather encourage us to pause before citing an 'official statistic' about our sector to make sure that we can be confident that it really is accurate, and in everyone's best interests to be using for the reasons we are.



UPDATE - 19 Sept 2018

There's been some new mapping research published today by Social Enterprise UK that seems to have picked up on some of these 'vagaries' I've outlined, and offers a more robust methodology in starting to scope the true shape and size of the social enterprise sector: https://www.socialenterprise.org.uk/the-hidden-revolution

Also, there's been some good debate and rebuttals and cross-rebuttals over on LinkedIn that you may also want to check out (including from the ex-deputy chief exec at Social Enterprise UK!): https://www.linkedin.com/feed/update/urn:li:activity:6447793971262418944

No comments:

Post a Comment