AIGM home  |  Our Community  |  About Us  | 
AIGM logo Australian Institute of Grants Management An OurCommunity.com.au enterprise


Forum    Login   Join

News

Are you measuring what matters?

By Chris Borthwick, thinker in residence, Our Community

This is a new data-driven era, and if you claim that your grants programs have had an impact, you have to back those claims with data. Anecdotes aren’t good enough. You need to set out your aims; provide a persuasive logic model suggesting the steps that should be taken to achieve those aims; map out the milestones that would tell you whether you were on track along the way; and compare actual performance with expected at every point.

You have to set out your criteria in advance, too, to avoid the temptation simply to draw bullseyes around wherever the arrows hit. You can’t argue with that.

But. There are more things in heaven and earth, Horatio, than are dreamt of in your criteria. The problem with measuring the outcomes you expect is that this doesn’t account for the unexpected – and it’s surprising how much of what happens in the course of work is surprising.

Let’s hop into our time machine and go back to 2003, when we’re a not-for-profit foundation listening to Mark Zuckerberg set out his metrics for the performance of his proposed new website, The Facebook. We’ll pretend for the moment that we’re not interested in the money (except insofar as we’re slightly worried about sustainability); as a foundation, we want to know what it will do to bring the community together.

Mr Zuckerberg sets out a number of hard targets. The new offering will allow people over the world to connect easily with their friends, and extend their friendship network by reaching out to their friends’ friends, and keep abreast of their family connections, creating a net of connectivity. He attaches numbers to these predictions – five million users by the end of five years. We give him the money he needs.

SqM

Our Community's Thinker in Residence Chris Borthwick considers good measurement.

And 15 years later we come back and see whether that was money well spent. According to our initial criteria, it certainly was. The number of users has gone up to 2.2. billion, and everything Zuckerberg said he would do he has done. No argument there.

However, it now also appears that in doing that he’s facilitated Russian interference in the American electoral process that may well have resulted in the election of an unfit leader to control the American nuclear arsenal. Among other things, Facebook has apparently facilitated genocide in Myanmar. And Zuckerberg has become one of the most powerful people in the world.

The initial criteria, it seems, didn’t capture everything important about the project.

As medical researchers say, there is no such thing as a side-effect. There are only effects – some that you expected, some that you didn’t; some that you want, some that you don’t. And the same applies to impact.

It’s not all bad. Things can go better than expected (don’t laugh, it can happen). Your work may throw off new discoveries or serendipitous breakthroughs. If you’re funding research, it’s supposed to – even, perhaps, expected to. If it’s research, though, it is by definition looking for things we don’t already know, and that we can’t in advance know are knowable. And even where what we do isn’t defined as research, everything we do should include a research element, seeking to diminish the sphere of ignorance, shedding light into dark corners.

Grant Makers Intelligence

So how do we square the circle? How do we assess the impact of what we didn’t think was going to happen? That has to involve moving beyond the numbers. Qualitative measurements can be as important as quantitative. Canadian health promoter Ron Labonte has sketched out an assessment procedure based on group dialogue around case studies. It’s resource-intensive, but it offers a pathway to what we actually want – not simply facts, but also meaning.

Whether or not you follow Labonte’s methods, your assessment plan should still include an opportunity for reflection on what actually happened and what in that was important. Assessment should provide you with a torch, not with blinkers. We need the wisdom of hindsight.

News
A better measure of value: The Universal Commons

Leading Australian philanthropist Alan Schwartz is tackling one of the hardest challenges the planet faces: to put a true value on the social and natural capital of the world, including health, literacy, trust, clean water and biodiversity.

News
Whatever happened to evidence-based policy making?

An abridged version of Gary Banks' address for the Alf Rattigan Lecture for the Australia and New Zealand School of Government (ANZSOG) that points the way for what's worked in the past, and what can be done to avoid policy on the run.

News
How evaluation can kill: Why we can't just keep evaluating to justify

Leading social impact thinker Ross Wyatt says many funders and grantseekers are trapped by evaluations aiming to prove what they did was right. Here's how to do better.

News
Cracking the code: Funders the key to unlocking evidence-based practice

Our Community's Chaos Controller and executive director Kathy Richardson examines how we might create a sector where there are incentives for using evidence.