There is a growing discussion among social impact organizations and those who fund them about how to measure impact. It is indeed a very slippery endeavor.
Mario Marino, Chairman of Venture Philanthropy Partners (a venture philanthropy fund in Washington D.C. that makes growth capital investments in nonprofits) has been encouraging nonprofits to measure outcomes for years. Indeed one of the fundamental characteristics of venture philanthropy is a reliance on metrics and outcomes for investment to happen. He recently wrote a post arguing that he is “increasingly worried that the vast majority of funders and nonprofits are achieving, at best, marginal benefit from their efforts to implement outcomes thinking.” He argues that in an zealous pursuit of metrics we have left common sense and “softer” impact behind and encouraged nonprofits to move away from the impact they were working towards.
To add further confusion to the outcome measurement discussion, the Gates Foundation’s Melinda Tuan studied 8 approaches to measuring cost vs. social impact, or the value that nonprofit organizations create versus the cost of their activities. The results of the study were disheartening; none of the approaches they studied was a magic bullet, all had significant drawbacks, which led them to conclude: “Integrated cost approaches to measuring and/or estimating social value are still in the nascent stages of development due to the lack of maturity in the field of social program evaluation.”
And there are other camps working towards outcome measurement, like those debating about whether randomized control trials (a research methodology where a random group of program participants is tracked and compared to a random group of cohorts who did not participate in the program) are feasible for nonprofits. And on the social business side, the GIIN (Global Impact Investing Network) is developing standards for measuring and communicating the social impact of investments known as The Impact Reporting and Investment Standards (IRIS). And that’s just a start.
This whole social impact measurement endeavor is incredibly important because if we can figure out a way to measure which social change efforts work, and which don’t, we can allocate resources accordingly and, in theory, get closer to solutions to social problems.
But I think we need to first take a step back. As is so often the case in efforts to build nonprofit capacity, effectiveness and infrastructure (including, in this case, the ability of nonprofits to evaluate their work) the focus is on the largest, most resourced nonprofit organizations. Let’s remember that more than 80% of nonprofit organizations have budgets under $1 million (see the Nonprofit Almanac). Budgets that small leave very little room for funds to support randomized control trials or other kinds of outcome measurements.
But an even bigger roadblock is the fact that many nonprofit organizations have not articulated their theory of change, or their logic model. Many nonprofit organizations are doing good work, but they don’t necessarily have an articulated strategy around that good work. A logic model helps an organization understand and articulate how they believe that they translate resources (inputs) into social impact, or change in a community. This understanding allows the organization to better articulate (to potential funders, volunteers, supporters, partners), and create strategy around, their work. A potential logic model for an English as a Second Language after-school program could be as follows:
One of the first steps Social Velocity undertakes with clients who want to increase organization capacity, sustainability, revenue, growth, or really any kind of progress, is to create a logic model with the organization. The majority of nonprofits that I encounter don’t have an articulated logic model or theory of change. It may seem like an academic exercise, but I would argue that it is absolutely critical to just about anything a nonprofit does. In order to understand their place in the community, the value that their work adds, how additional inputs (like funding) can increase impact, and their strategy for delivering services, they need to articulate this process.
But the larger debate about outcome measurement ignores the fact that the majority of nonprofit organizations have not completed step 1 in outcome measurement: articulating a strategy for using resources to create outcomes. Once this is articulated, we can talk about how to measure whether that strategy is actually coming to fruition.
Nell,
Excellent points! If we can get a substantial number of nonprofits to go through a theory of change process, we should have a major celebration!
Ken Berger
Charity Navigator
Thanks Ken! Better yet, if we could get philanthropists to fund a theory of change process for the nonprofits they support that would help to ensure that it happens.
Yes Nell, there are a lot of people keen to do things about outcomes in philanthropy and not-for-profits and lots of people trying very hard to develop frameworks and approaches. I am with you regarding everything you have said in your blog posting here. But as you say, it can be complex.
As you say, the key to it all is the development of a good theory of change or logic model. The area of logic modeling or what is also termed ‘outcomes modeling’ is moving into hyperdrive at the moment. Where I am wanting to take this is to build on the insight that if you draw your logic models (outcomes models) in the right way (and this is now moving on just a little from the traditional one page inputs, outputs in the diagram shown in your posting) you can then use such models for a whole range of different outcomes-related organization purposes. (See here).
Just to show you an example, here is an outcomes model for a mocked-up Sustainable Houses Project which starts to show how it can be used for all the things related to dealing with outcomes. here. There are more of these at OutcomesModels.org.
This way of working, if you work through the whole approach, lets you deal with all of the technical issues around working with outcomes (e.g. how to deal with non-measurable outcomes, how to do outcomes-focused contracting etc.). I’ve written a paper on building a monitoring and evaluation system based on this approach which includes how-to videos and resources. <a href="http://knol.google.com/k/m-e-systems-how-to-build-an-affordable-simple-monitoring-and-evaluation-systemhere".here. This approach is also known as the Easy Outcomes Approach.
Thanks for your great post.
Paul Duignan, PhD (Follow me on OutcomesBlog.org, Twitter or my E-Newsletter on Outcomes and Evaluation.
Paul,
Thanks for the list of tools. This is very helpful. It looks like you are doing great work. Thanks for writing.
Before I start I will confess that I am big proponent of modeling be it program or theory. I have yet to meet a client, when engaged in this process fully, has not come out on the other side as clearer and more focused. What however it most catalyzing for them is not the model itself but the process of modeling which we describe as the systematic articulation, reflection, critique and refinement of their intent including strategies and underlying principles. The “model” itself is just a bonus.
It is not a silver bullet for what ails the sectors but it can bring to light, and thus make more transparent, what we hope will be be achieved for our efforts. After that, to be honest, the evaluative piece, which in our opinion is merely a focus on “to what end” is much easier to determine. Funding it however is another issue.
Jara,
I completely agree that the process of modeling is really the key part. If a nonprofit can come together and agree about what it is that they are trying to do, the impact they are trying to achieve, that is huge.
And I also agree that funding the evaluation, or even the creation of the logic model, is a critical hurdle. Most foundations and donors want their money to go to “program” as opposed to planning, strategy, infrastructure, etc. But I think that is changing. And so I believe that soon there will be more money for this type of planning and the resulting evaluation of the social impact nonprofits are working toward.
Neil,
I too think the tide is turning. I have always been challenged by the notion that things like strategy, program/implementation and evaluative thinking are separate rather than threads when woven together are the fabric of success.
Nell,
I agree completely that most nonprofits haven’t articulated their theory of change.
I am finding the theory of change missing, however, from the logic model that you are using. I would add a few columns at the beginning:
1. What is the problem/need, it’s scope and root causes
2. What is the Theory of Change that will get from from the problem/need, given it’s root causes, to the outcomes and impact that you desire? Theory of change includes not only evidence based practice, but also experimental models based on the root causes, and, something we forget to talk about, the mental models and political theory about how the world works… I haven’t met an organization that is immune from that. Embedded in those mental models are also are values systems.
3. Once defined, then what is critical in completing the logic model, before you get to inputs, is the “strategic” part, or what can also be thought of as “where is the leverage” This is essential to defining the path your organization will follow as there are way more choices than there are resources available to any one nonprofit.
Best, Gayle
Gayle,
You variation on the logic model is interesting and definitely much deeper than the model I suggest. I think it would be great if the average nonprofit organization could create a logic model like the one you describe, but I worry that it is too complex and has too many steps to work for the majority of nonprofits. I wonder if it is better to at least start with a simpler version that gets to the core argument they are making in their organization (how they use resources to create outcomes), and then the more sophisticated nonprofits could delve deeper with a model like the one you suggest.
Nell, again. Excellent post. However, it seems you’ve glossed over an important question: who will or should pay for impact assessment? As you know this is a debate that was alive in well in the philanthropic sector about a decade ago. I’m actually shocked we don’t see more references to this in the current dialogue on the topic. The primary issue does not appear to be about capacity, but funding. While foundation’s have paid tremendous amounts of lip service to measuring impact over the years, very few have shown a willingness to fund these activities in a substantial manner.
With respect to social enterprises, who should bear the cost of impact assessment? Companies, investors, clients/customers? In light of the current challenges facing the social venture space, none of these potential actors seem to fit the bill. Would very much enjoy your thoughts on this.
David,
I think you are absolutely correct, we need to figure out who will pay for assessment. But, really, the bigger question is who will pay for infrastructure, capacity, R&D, etc., all of those things that are routinely paid for in the for-profit sector, but very difficult to fund in the social impact space. However, I see this trend changing as (mostly) foundations and individual major donors are realizing that they can get a much higher return on their investment when they invest in smart infrastructure, capacity, assessment, planning, etc. I think the change here will be by example, not by some big entity stepping up and starting to fund these things. But nonetheless, change is coming.