AIGM home  |  Our Community  |  About Us  | 
AIGM logo Australian Institute of Grants Management An OurCommunity.com.au enterprise


Forum    Login   Join

News

Are you measuring what matters?

By Chris Borthwick, thinker in residence, Our Community

This is a new data-driven era, and if you claim that your grants programs have had an impact, you have to back those claims with data. Anecdotes aren’t good enough. You need to set out your aims; provide a persuasive logic model suggesting the steps that should be taken to achieve those aims; map out the milestones that would tell you whether you were on track along the way; and compare actual performance with expected at every point.

You have to set out your criteria in advance, too, to avoid the temptation simply to draw bullseyes around wherever the arrows hit. You can’t argue with that.

But. There are more things in heaven and earth, Horatio, than are dreamt of in your criteria. The problem with measuring the outcomes you expect is that this doesn’t account for the unexpected – and it’s surprising how much of what happens in the course of work is surprising.

Let’s hop into our time machine and go back to 2003, when we’re a not-for-profit foundation listening to Mark Zuckerberg set out his metrics for the performance of his proposed new website, The Facebook. We’ll pretend for the moment that we’re not interested in the money (except insofar as we’re slightly worried about sustainability); as a foundation, we want to know what it will do to bring the community together.

Mr Zuckerberg sets out a number of hard targets. The new offering will allow people over the world to connect easily with their friends, and extend their friendship network by reaching out to their friends’ friends, and keep abreast of their family connections, creating a net of connectivity. He attaches numbers to these predictions – five million users by the end of five years. We give him the money he needs.

SqM

Our Community's Thinker in Residence Chris Borthwick considers good measurement.

And 15 years later we come back and see whether that was money well spent. According to our initial criteria, it certainly was. The number of users has gone up to 2.2. billion, and everything Zuckerberg said he would do he has done. No argument there.

However, it now also appears that in doing that he’s facilitated Russian interference in the American electoral process that may well have resulted in the election of an unfit leader to control the American nuclear arsenal. Among other things, Facebook has apparently facilitated genocide in Myanmar. And Zuckerberg has become one of the most powerful people in the world.

The initial criteria, it seems, didn’t capture everything important about the project.

As medical researchers say, there is no such thing as a side-effect. There are only effects – some that you expected, some that you didn’t; some that you want, some that you don’t. And the same applies to impact.

It’s not all bad. Things can go better than expected (don’t laugh, it can happen). Your work may throw off new discoveries or serendipitous breakthroughs. If you’re funding research, it’s supposed to – even, perhaps, expected to. If it’s research, though, it is by definition looking for things we don’t already know, and that we can’t in advance know are knowable. And even where what we do isn’t defined as research, everything we do should include a research element, seeking to diminish the sphere of ignorance, shedding light into dark corners.

So how do we square the circle? How do we assess the impact of what we didn’t think was going to happen? That has to involve moving beyond the numbers. Qualitative measurements can be as important as quantitative. Canadian health promoter Ron Labonte has sketched out an assessment procedure based on group dialogue around case studies. It’s resource-intensive, but it offers a pathway to what we actually want – not simply facts, but also meaning.

Whether or not you follow Labonte’s methods, your assessment plan should still include an opportunity for reflection on what actually happened and what in that was important. Assessment should provide you with a torch, not with blinkers. We need the wisdom of hindsight.

Wire
Grantmaking in Australia conference 2018

The 2018 Grantmaking in Australia Conference will connect you with the people who have stared down the challenges and give you the confidence you need to take one small step for your organisation, towards one giant leap for humankind.

Prepare for impact: Suit up, strap and count down for better grantmaking. Save the date for our conference now.

News
How are you measuring your social impact?

There are some big players in the world of social impact reporting, with think tanks and technology companies vying to solve one of the toughest problems we face: how can organisations measure their social worth?

News
Are you measuring what matters?

This is a new data-driven era, and if you claim that your grants programs have had an impact, you have to back those claims with data. Anecdotes aren't good enough.

News
Meet the trainer: Gina Cirillo

How our SmartyGrants trainer can "save your (grantmaking) life".