Follow Social Velocity on Google Plus Follow Social Velocity on Facebook Follow Nell Edgington on Twitter Follow SocialVelocity on Linked In View the Social Velocity YouTube Channel Get the Social Velocity RSS Feed

Download a free Financing Not Fundraising e-book when you sign up for email updates from Social Velocity.

Let’s Take a Step Back in the Outcomes Debate

By Nell Edgington



There is a growing discussion among social impact organizations and those who fund them about how to measure impact.  It is indeed a very slippery endeavor.

Mario Marino, Chairman of Venture Philanthropy Partners (a venture philanthropy fund in Washington D.C. that makes growth capital investments in nonprofits) has been encouraging nonprofits to measure outcomes for years.  Indeed one of the fundamental characteristics of venture philanthropy is a reliance on metrics and outcomes for investment to happen.  He recently wrote a post arguing that he is “increasingly worried that the vast majority of funders and nonprofits are achieving, at best, marginal benefit from their efforts to implement outcomes thinking.”  He argues that in an zealous pursuit of metrics we have left common sense and “softer” impact behind and encouraged nonprofits to move away from the impact they were working towards.

To add further confusion to the outcome measurement discussion, the Gates Foundation’s Melinda Tuan studied 8 approaches to measuring cost vs. social impact, or the value that nonprofit organizations create versus the cost of their activities.  The results of the study were disheartening; none of the approaches they studied was a magic bullet, all had significant drawbacks, which led them to conclude: “Integrated cost approaches to measuring and/or estimating social value are still in the nascent stages of development due to the lack of maturity in the field of social program evaluation.”

And there are other camps working towards outcome measurement, like those debating about whether randomized control trials (a research methodology where a random group of program participants is tracked and compared to a random group of cohorts who did not participate in the program) are feasible for nonprofits. And on the social business side, the GIIN (Global Impact Investing Network) is developing standards for measuring and communicating the social impact of investments known as The Impact Reporting and Investment Standards (IRIS).  And that’s just a start.

This whole social impact measurement endeavor is incredibly important because if we can figure out a way to measure which social change efforts work, and which don’t, we can allocate resources accordingly and, in theory, get closer to solutions to social problems.

But I think we need to first take a step back.  As is so often the case in efforts to build nonprofit capacity, effectiveness and infrastructure (including, in this case, the ability of nonprofits to evaluate their work) the focus is on the largest, most resourced nonprofit organizations.  Let’s remember that more than 80% of nonprofit organizations have budgets under $1 million (see the Nonprofit Almanac).  Budgets that small leave very little room for funds to support randomized control trials or other kinds of outcome measurements.

But an even bigger roadblock is the fact that many nonprofit organizations have not articulated their theory of change, or their logic model.  Many nonprofit organizations are doing good work, but they don’t necessarily have an articulated strategy around that good work.  A logic model helps an organization understand and articulate how they believe that they translate resources (inputs) into social impact, or change in a community.  This understanding allows the organization to better articulate (to potential funders, volunteers, supporters, partners), and create strategy around, their work.  A potential logic model for an English as a Second Language after-school program could be as follows:

One of the first steps Social Velocity undertakes with clients who want to increase organization capacity, sustainability, revenue, growth, or really any kind of progress, is to create a logic model with the organization.  The majority of nonprofits that I encounter don’t have an articulated logic model or theory of change.  It may seem like an academic exercise, but I would argue that it is absolutely critical to just about anything a nonprofit does.  In order to understand their place in the community, the value that their work adds, how additional inputs (like funding) can increase impact, and their strategy for delivering services, they need to articulate this process.

But the larger debate about outcome measurement ignores the fact that the majority of nonprofit organizations have not completed step 1 in outcome measurement: articulating a strategy for using resources to create outcomes.  Once this is articulated, we can talk about how to measure whether that strategy is actually coming to fruition.


Learn more about nonprofit innovation and
download a free Financing Not Fundraising e-book
when you sign up for email updates
from Social Velocity.


About the Author: Nell Edgington is President of Social Velocity (www.socialvelocity.net), a management consulting firm leading nonprofits to greater social impact and financial sustainability. Social Velocity helps nonprofits grow their programs, bring more money in the door, and use resources more effectively. For more information, check out Social Velocity consulting services and clients.

Thursday, January 21st, 2010 Capacity Building, Nonprofits, outcomes, Planning, Strategy

Tags: , , , , , , , , , , , , ,


14 Comments to Let’s Take a Step Back in the Outcomes Debate

Ken Berger
January 21, 2010

Nell,

Excellent points! If we can get a substantial number of nonprofits to go through a theory of change process, we should have a major celebration!

Ken Berger
Charity Navigator

Nell Edgington
January 21, 2010

Thanks Ken! Better yet, if we could get philanthropists to fund a theory of change process for the nonprofits they support that would help to ensure that it happens.

Paul Duignan
January 22, 2010

Yes Nell, there are a lot of people keen to do things about outcomes in philanthropy and not-for-profits and lots of people trying very hard to develop frameworks and approaches. I am with you regarding everything you have said in your blog posting here. But as you say, it can be complex.

As you say, the key to it all is the development of a good theory of change or logic model. The area of logic modeling or what is also termed ‘outcomes modeling’ is moving into hyperdrive at the moment. Where I am wanting to take this is to build on the insight that if you draw your logic models (outcomes models) in the right way (and this is now moving on just a little from the traditional one page inputs, outputs in the diagram shown in your posting) you can then use such models for a whole range of different outcomes-related organization purposes. (See here).

Just to show you an example, here is an outcomes model for a mocked-up Sustainable Houses Project which starts to show how it can be used for all the things related to dealing with outcomes. here. There are more of these at OutcomesModels.org.

This way of working, if you work through the whole approach, lets you deal with all of the technical issues around working with outcomes (e.g. how to deal with non-measurable outcomes, how to do outcomes-focused contracting etc.). I’ve written a paper on building a monitoring and evaluation system based on this approach which includes how-to videos and resources. <a href="http://knol.google.com/k/m-e-systems-how-to-build-an-affordable-simple-monitoring-and-evaluation-systemhere&quot;.here. This approach is also known as the Easy Outcomes Approach.

Thanks for your great post.

Paul Duignan, PhD (Follow me on OutcomesBlog.org, Twitter or my E-Newsletter on Outcomes and Evaluation.

Nell Edgington
January 25, 2010

Paul,

Thanks for the list of tools. This is very helpful. It looks like you are doing great work. Thanks for writing.

Jara Dean-Coffey
January 26, 2010

Before I start I will confess that I am big proponent of modeling be it program or theory. I have yet to meet a client, when engaged in this process fully, has not come out on the other side as clearer and more focused. What however it most catalyzing for them is not the model itself but the process of modeling which we describe as the systematic articulation, reflection, critique and refinement of their intent including strategies and underlying principles. The “model” itself is just a bonus.

It is not a silver bullet for what ails the sectors but it can bring to light, and thus make more transparent, what we hope will be be achieved for our efforts. After that, to be honest, the evaluative piece, which in our opinion is merely a focus on “to what end” is much easier to determine. Funding it however is another issue.

Nell Edgington
January 27, 2010

Jara,

I completely agree that the process of modeling is really the key part. If a nonprofit can come together and agree about what it is that they are trying to do, the impact they are trying to achieve, that is huge.

And I also agree that funding the evaluation, or even the creation of the logic model, is a critical hurdle. Most foundations and donors want their money to go to “program” as opposed to planning, strategy, infrastructure, etc. But I think that is changing. And so I believe that soon there will be more money for this type of planning and the resulting evaluation of the social impact nonprofits are working toward.

Jara Dean-Coffey
January 28, 2010

Neil,

I too think the tide is turning. I have always been challenged by the notion that things like strategy, program/implementation and evaluative thinking are separate rather than threads when woven together are the fabric of success.

Gayle L. Gifford, ACFRE
February 4, 2010

Nell,
I agree completely that most nonprofits haven’t articulated their theory of change.
I am finding the theory of change missing, however, from the logic model that you are using. I would add a few columns at the beginning:
1. What is the problem/need, it’s scope and root causes
2. What is the Theory of Change that will get from from the problem/need, given it’s root causes, to the outcomes and impact that you desire? Theory of change includes not only evidence based practice, but also experimental models based on the root causes, and, something we forget to talk about, the mental models and political theory about how the world works… I haven’t met an organization that is immune from that. Embedded in those mental models are also are values systems.
3. Once defined, then what is critical in completing the logic model, before you get to inputs, is the “strategic” part, or what can also be thought of as “where is the leverage” This is essential to defining the path your organization will follow as there are way more choices than there are resources available to any one nonprofit.

Best, Gayle

Nell Edgington
February 5, 2010

Gayle,

You variation on the logic model is interesting and definitely much deeper than the model I suggest. I think it would be great if the average nonprofit organization could create a logic model like the one you describe, but I worry that it is too complex and has too many steps to work for the majority of nonprofits. I wonder if it is better to at least start with a simpler version that gets to the core argument they are making in their organization (how they use resources to create outcomes), and then the more sophisticated nonprofits could delve deeper with a model like the one you suggest.

uberVU - social comments
February 18, 2010

Social comments and analytics for this post…

This post was mentioned on Twitter by aaronsklar: Many nonprofits’ lack of theory of change is a major hindrance for social impact measurement! http://twurl.nl/kdvziy #socent RT @sc_seo…

David
February 20, 2010

Nell, again. Excellent post. However, it seems you’ve glossed over an important question: who will or should pay for impact assessment? As you know this is a debate that was alive in well in the philanthropic sector about a decade ago. I’m actually shocked we don’t see more references to this in the current dialogue on the topic. The primary issue does not appear to be about capacity, but funding. While foundation’s have paid tremendous amounts of lip service to measuring impact over the years, very few have shown a willingness to fund these activities in a substantial manner.

With respect to social enterprises, who should bear the cost of impact assessment? Companies, investors, clients/customers? In light of the current challenges facing the social venture space, none of these potential actors seem to fit the bill. Would very much enjoy your thoughts on this.

Nell Edgington
February 22, 2010

David,

I think you are absolutely correct, we need to figure out who will pay for assessment. But, really, the bigger question is who will pay for infrastructure, capacity, R&D, etc., all of those things that are routinely paid for in the for-profit sector, but very difficult to fund in the social impact space. However, I see this trend changing as (mostly) foundations and individual major donors are realizing that they can get a much higher return on their investment when they invest in smart infrastructure, capacity, assessment, planning, etc. I think the change here will be by example, not by some big entity stepping up and starting to fund these things. But nonetheless, change is coming.

[...] bottom line as a barometer of success. Rather, a nonprofit must articulate what they exist to do. A theory of change, or logic model, allows a nonprofit to state (to internal board and staff, and to external funders, [...]

[...] Do we know if we are accomplishing anything? Because nonprofit organizations can’t simply look at a profit and loss statement to see progress, determining success is much more difficult than in the for-profit world. Yet a nonprofit organization cannot just translate community resources into activities and call it a day. Nonprofits are increasingly forced to demonstrate the change their work creates in the community. I’m not suggesting that every nonprofit must conduct large evaluation projects. Rather, I’m arguing that a nonprofit must create a solid strategy for creating change and then find a way (as cheaply and simply as possible) to determine whether they are delivering on that strategy. [...]

Leave a comment