Note: As I mentioned earlier, I am taking a few weeks away from the blog to relax and reconnect with the world outside of social change. But I am leaving you in the incredibly capable hands of a rockstar set of guest bloggers. Next up is Kelly Born, program officer at the Hewlett Foundation working on their Madison Initiative, which focuses on reducing today’s politically polarized environment. Kelly also writes for the always thoughtful Hewlett Foundation blog. Here is her guest post…
In March of 2014, the William and Flora Hewlett Foundation launched a new initiative focused on US democracy reform, The Madison Initiative. The overarching goal is to “help create the conditions in which Congress and its members can deliberate, negotiate, and compromise in ways that work for more Americans.”
Our mandate is for a 3-year, exploratory initiative to assess whether and how the Foundation might be able to make a difference here. During this period, we are focused on three central questions:
- Are there solutions and approaches that are worth pursuing?
- Is there ample grantee capacity to pursue these ideas (or can we help build it)?
- Are there funding partners we can work with to make it happen?
In exploring this problem of congressional dysfunction we realized early on that, unfortunately, there don’t appear to be any silver-bullets that will solve this problem – it’s not as if campaign finance reform, nonpartisan redistricting, or increased voter turnout, taken on their own, would resolve our current democratic ails (even setting aside for the moment how hard it would be to actually achieve these changes!).
Regrettably, there is no clear consensus on what to do to improve the system, much less on how to do it. This may be, in part, why Inside Philanthropy awarded The Madison Initiative with 2014’s Big Foundation Bet Most Likely to Fail. Given this, our view has been that current congressional dysfunction is occurring in a system of systems (and sub-systems) that are interacting in complicated ways.
Early on we decided to develop a systems map rather than a theory of change to guide our work (working in close partnership with the Center for Evaluation Innovation and Kumu, collaborations we’ve written a bit about here). Theories of change typically outline desired (social or environmental) outcomes and then map backwards, linearly, to the activities and inputs necessary to achieve those outcomes. Systems maps are perhaps better suited for more complex, uncertain environments like democracy reform, where cause-and-effect relationships can be entangled and mutually reinforcing, rather than unidirectional.
Version 1.0 of our map includes more than 35 variables we believe are contributing to the problem, distributed across three key domains: Congress, Campaigns and Elections, and Citizens. In light of this complexity, rather than making an initial set of big bets on a few key variables, we have instead spread a series of smaller bets within these systems to see where grantees might gain traction, and what this reveals about the system’s more confounding parts.
The benefits of this approach are many – in fact, I cannot imagine effectively tackling this particular problem any other way. But employing this spread betting approach also involves a few challenges for us at Hewlett, and for our partners and grantees. The trade-offs are worth considering:
- We are acknowledging and respecting complexity, but this can sow seeds of confusion for our partners. Our approach has the essential benefit of taking into account the systemic complexity and interdependency of what we are trying to help change. We are avoiding over-simplifying and thereby misconstruing our reality (a good thing). But we are exploring more than 35 variables (ranging from deteriorating bipartisan relationships to the proliferation of partisan news media), with more than 60 active grantees. This approach can be hard to manage, and harder still to convey to others – especially anyone accustomed to a more linear and readily understandable theory of change.
- Our course correcting helps us learn, but has a real impact on partners. As we diversify our investments to learn more about what works, we will continue to learn more about which efforts are having the most impact on congressional dysfunction, and which are less germane to the problem. As we do, we will necessarily converge (and double down) on a few core interventions, while discontinuing others. This will mean disappointing organizations that we respect and had supported at the outset – an inevitable byproduct of this approach, but unpleasant for all involved.
- Our evidence-based approach risks coming off as overly academic. We are determined to avoid investing in solutions where there is not solid evidence to support their viability vis-à-vis our goals. This helps us avoid squandering funds on interventions that won’t, ultimately, work. But this approach also runs the risk of coming across as standoffish, academic, and idiosyncratic in the eyes of a practitioner-driven field that in some instances may be pursuing work that is harder to (or has yet to be) substantiated by solid research.
We’ve certainly got our work cut out for us. But we deeply believe that the social sector shouldn’t shy away from complex problems. We also believe that the benefits of this approach far outweigh the costs. It enables broad-based learning, and truly forces us to constantly re-think the grants we are making. Building in these tough choices, rather than forging ahead with a pre-defined strategy, requires that we not just learn, but that we act on what we discover. And fast.
In short, while beset by a few real challenges, we’re convinced that an emergent path is the best path forward. Surely we will place some wrong bets along the way. But, as a favorite colleague of mine often says, “it’s not like we’re selling cigarettes to children.” All of our grantees are doing great work – ultimately it will (not so simply) be a question of which of these lines of work is most likely to improve Congress.
In 2017, we will go back to our Board of Directors to discuss whether and how The Madison Initiative’s work will continue. In the meantime, we would love to hear how other funders have approached emergent problems like this – and how nonprofits might advise that we manage these inherent challenges as we progress?
Note: As I mentioned earlier, I am taking a few weeks away from the blog to relax and reconnect with the world outside of social change. But I am leaving you in the incredibly capable hands of a rockstar set of guest bloggers. First up is David Henderson, Director of Analytics for Family Independence Initiative, a national nonprofit which leverages the power of information to illuminate and accelerate the initiative low-income families take to improve their lives. David also writes his own blog, Full Contact Philanthropy, which is amazing. Here is his guest post…
In early June I was invited to be on a data mining panel at the Stanford Social Innovation Review Data on Purpose conference. The conference was full of nonprofit executives interested in tapping the big data revolution for social good. Naturally, the panel moderator asked us panelist to weigh in on if, and how, data was changing the social sector. Characteristically, I turned a feel-good question into a critique of the state of analytics in the social sector, which I’ve written about elsewhere and will expand on here.
Data is not changing the social sector. I would argue it’s not changing the world either. While it is very likely that data is changing your world, I do not believe data is changing the world.
For all the talk about how data is revolutionizing the world and that software is eating everyone’s lunch, the fact is that for the over two billion people who have no lunch to eat (literally and figuratively), the impact of the data revolution is muted, if nonexistent all together. Changing the world indeed.
The corporate data revolution has largely been fueled by data exhaust. Data exhaust is comprised of the various digital breadcrumbs you and I leave all over the Internet but that we might not think about as data in a traditional sense. For example, companies like Facebook and Amazon don’t simply log data when you click “submit”, they track your every movement around the Internet, logging every click and clack, allowing unprecedented marketing optimization. All these additional metrics are data exhaust, as consumers are almost passively generating data marketers can capture and monetize for almost nothing.
On the social sector data conference circuit, countless data-wonk hopefuls mindlessly espouse all the incredible things nonprofits can do now that data acquisition costs have been driven almost to zero. This is nonsense, as the social sector has no such data exhaust analogue, which is why the social sector doesn’t truly have big data.
Nonprofits often work with populations with a number of barriers, which drives up the cost of data acquisition relative to for-profit counterparts. Just some of the data collection barriers nonprofits grapple with include working with populations with low levels of literacy or limited to no access to technology. How exactly is one going to generate digital exhaust without any digital possessions in the first place, or while working three jobs to support her family?
Obviously, you don’t. The barriers too many people face in this world are exactly why nonprofits are in the business of social change in the first place. But it is also why we are so poorly poised to capitalize on the alleged data ubiquity, as that revolution is not permeating class boundaries to the extent technology evangelists would have us believe.
Another reason why data is not changing the world, or rather, why the social sector is failing to change the world with data, is that by and large we simply are not investing in the necessary capacity to turn data into insights.
While a new “data for the social sector” company with an unfortunate misspelling of a common word seems to pop up every day, there are very few companies actually building the tools the sector needs to put data in to action. Meanwhile, our technological overlords in Silicon Valley are depressingly stuck on the assumption that innovation in the social sector means fundraising software. Sigh.
If we want to use data to change the world, we need to think beyond software tools and simple (if colorful) data visualizations. Nonprofits need to invest in building their own analytical capacity, both by hiring analysts and also by investing in the entire staff’s ability to be intelligent consumers of data analysis.
Illusion of Insight
Everyone loves the idea of being data driven, but very few organizations actually want to make the investment. My employer, the Family Independence Initiative (FII), did make that investment. In turn, FII is now able to not only run regressions and build decision tree models, but can continuously learn from its data, augmenting every level of the organization from Chief Executive to line staff.
That investment is not cheap. Worse yet, like any good analyst, I can be a major buzz-kill. Much of my time is spent explaining why a particular regression coefficient doesn’t necessarily mean we are super awesome. In fact, a good analyst can make you less sure of your social impact.
But facing the tough reality paves the way to real impact. We cannot collectively do more without exactingly quantifying how little we’ve accomplished. These are tough truths, and most nonprofits would rather assume the hypothesis of their greatness, leaving no room for data’s insights.
The Path Forward
Just because data is not changing the world does not mean data cannot change the world. I believe it can, which is why I do what I do. While by and large nonprofits fail to invest in rigorous analysis, organizations like GiveDirectly are leading by example, showing what is possible when fact is paramount to fundraising.
Ultimately, being data driven is less about statistical techniques and more about a relentless commitment to the truth. The truth is that data is not changing the world. But if we, as a sector, can elevate the truth above all else, then we might just be able to change the world after all.
Tris is Director of Development for New Philanthropy Capital (NPC), a U.K. think tank and consultancy that works with both nonprofits and funders. Tris focuses on both the demand and supply sides of innovation around social impact. His particular interest is putting impact at the heart of the social sector, including shared measurement, open data and systems thinking. He helped initiate, and now coordinates, the Inspiring Impact program which aims to embed impact measurement across the UK charity sector by 2022. He is also a trustee of the Social Impact Analysts Association, a member of the EU GECES subgroup on impact measurement in social enterprise, and the Leap of Reason Ambassadors Community.
Nell: A big focus of your work at NPC is making impact measurement ubiquitous in the UK’s nonprofit sector. How far is there to go and how does the UK compare to the US in impact measurement being a norm?
Tris: There’s undoubtedly been significant progress over the last decade on impact measurement in the UK, and NPC has been at the heart of that. There are several ways in which that progress is visible, as well as in the sector level surveys NPC has done to track change. For example, most charities say that they have invested more in impact measurement in the last five years, and as a result we see that it is increasingly the norm for charities to have a defined theory of change, a role within the organisation to lead on impact measurement, and to talk about their impact measurement efforts in their public reporting. Most institutional funders also say that they look for evidence of charities’ impact measurement efforts in their funding decisions. Demand for measurement advice is growing, and the impact measurement industry is growing in response – there are more consultants offering services in this area.
The growth of social (or impact) investing has also driven greater interest in impact measurement. The industry as a whole acknowledges the centrality of impact measurement and the need for social returns to be as well evidenced as financial returns. There have been a number of key developments to move the field forward here, from Big Society Capital’s outcomes matrix to the G8 Social Impact Investment Taskforce and European GECES reports and guidance on impact measurement – all of which NPC has helped to deliver.
What’s not as clear is how much progress there’s been on the use of impact measurement, rather than its mere existence. When NPC repeats our field level state of the sector research in 2016, we’ll be asking a number of questions to tease out whether impact measurement activity is leading to use of impact evidence in decision-making – whether it’s becoming embedded in practice.
My concern is that we don’t see the signs that impact measurement is driving learning, improvement, decision-making or wholesale shifts in allocating resources towards higher impact interventions, programmes and organisations. It feels like impact measurement is something that everyone acknowledges we need to do, but few have worked out how to use. With the result that it’s bolted on to the reality of organisations delivering services and raising funding, but not embedded at the core.
A few examples of what I mean: if impact measurement were driving learning, I’d expect to see lots of organisations sharing their insights on success and failure, and learning from each other. I’d expect to see common measurement frameworks which allow organisations to understand their relative performance. These are still very rare. I’d also expect to see investment by funders and investors in the infrastructure that we know is needed for learning – journals, online forums and repositories and practitioner networks. There are some emerging examples of these, like the What Works Centres, but they’re still mostly just getting off the drawing board.
Most importantly I’d expect to see charities adjusting strategies and programmes in response to their learning. Maybe I’m not looking in the right places, but the examples I do see are the exception, not the norm.
When it comes to comparing the UK and US, it’s really hard. We don’t have comparable field-level studies, and we need to work together more closely on these if we want robust insights. For example, if you compare the findings in NPC’s 2012 paper with a recent US study it looks like nonprofits are more likely to say the main purpose of impact measurement is learning and improvement. But actually we don’t know if this is the result of the questions we asked and how we asked them.
In both the US and the UK, it’s clear that the rhetoric on impact measurement has advanced over the last decade. What’s not yet clear is how the reality underlying that has shifted.
Nell: While there are many similarities between the US and UK nonprofit sectors there are some fundamental differences, in particular views about how much government (vs. private charity) should do for public welfare. How does the UK’s view of government’s role help or hurt the capacity building efforts of nonprofits?
Tris: The UK government has taken on a leading role in the social investment space, and it’s here that efforts to build capacity are most visible. Investment readiness programmes have been introduced over the past few years to build general capacity to access social investment. More recently, impact readiness programmes have arrived to do the same for impact measurement capacity. NPC has been working within these programmes to help a number of charities, and cohorts of charities, and it’s clear that they can play a major role in helping the sector to improve. But capacity-building in general has felt the effects of austerity just as much as any other area of government funding. Perhaps more so, as limited funds are increasingly focused on service delivery, not on efforts to improve services.
When NPC repeats its survey of the field, I am certain that we’ll find that limited funding to develop impact measurement capacity is still the major barrier cited by charities. It doesn’t look like anything’s going to change that any time soon.
Nell: NPC works at the nexus between nonprofits and funders, helping the two groups to understand and adopt impact measurement. In the US few funders will fund impact measurement systems, even though they want the data. How does NPC work to convince funders of the need for investments in measurement (among other capacity building investments)? What progress have you seen and what’s necessary for similar progress to happen in the US?
Tris: While a proportion of funders have for a long time supported evaluation, the majority still don’t. We’ve worked through programmes like Inspiring Impact (a sector-level collaborative programme to help embed impact measurement) with a group of funders to develop principles, and help them to embed support for impact measurement in their practice. These efforts can help those who already see the benefit of capacity-building to advance their work, but it’s tough to engage those who aren’t already thinking in this way. I think that the leap we need to make is to selling impact measurement through its benefits, by showing how organisations improve, and their impact increases, as a result. And because impact measurement isn’t yet typically embedded in organisations, those benefits aren’t as evident as they should be.
What does seem to work well is trying to get funders and charities to work together in a specific outcome area to make progress, rather than making a general case for impact measurement. Cohort capacity-building programmes, learning forums and shared measurement initiatives are all part of this. The key thing here is that then the funder is committed to the outcomes everyone’s working towards, and impact measurement becomes a tool for everyone to achieve those outcomes together.
Nell: You are part of the Leap Ambassador Community that recently released the Performance Imperative. Have you seen similar interest groups forming around these issues in the UK? And what role do you think interest groups like these play in a norm shift for the sector?
I have been privileged to be part of this amazing community of leaders, and one of a minority initially from outside the US. I’m convinced we need a similar movement here in the UK, and globally and have been discussing whether and how to approach this with the group from the start. And as co-Chair of Social Value International – a network of those working in the social impact field, I’m part of an effort to do this at the practitioner level too.
The Leap Ambassadors Community brings a human face to what is often seen as a technical subject. After 11 years of working in the social impact field, I am convinced that we cannot sell impact measurement just by increasing the supply of good technical solutions. We need a movement to build the demand for those solutions. We need the right frameworks to measure impact and manage performance. But we need the leaders to demand them, and to harness them to hold themselves accountable, learn and improve, and share what they find.
Photo Credit: NPC
This spring I have been trumpeting the Performance Imperative, a detailed definition of a high-performing nonprofit released by the Leap Ambassador community in March. Today I continue the ongoing blog series describing each of the 7 Pillars of the Performance Imperative with Pillar #2: Disciplined, People-Focused Management.
With this second Pillar, the Performance Imperative obviously makes a distinction between “leaders” in Pillar 1, and “managers” in Pillar 2. There is a note in the Performance Imperative that “leaders” and “managers” are typically two separate people in nonprofits with budgets over $1 million. So this distinction, and perhaps this Pillar, applies only to larger nonprofits.
But I think there is actually application to any nonprofit. In any nonprofit there are leadership tasks (creating the vision, being the cheerleader, marshaling resources) and there are management tasks (making sure the trains run on time, putting each resource to its highest and best use). In smaller organizations both sets of tasks fall to the same person, yet they both still need to be performed well. So I think it behooves any size nonprofit to analyze whether they are BOTH leading and managing well.
Effective managers put organization resources to their highest and best use. They recruit, train and retain the right talent, they use data to make good decisions, they manage to performance, and they are accountable.
You can read a larger description of Pillar 2 in the Performance Imperative, but here are some of the characteristics of a nonprofit that exhibits Disciplined, People-Focused Management:
- Managers translate leaders’ drive for excellence into clear workplans and incentives to carry out the work effectively and efficiently.
- Managers…recruit, develop, engage, and retain the talent necessary to deliver on the mission.
- Managers provide opportunities for staff to see…how each person’s work contributes to the desired results.
- Managers establish accountability systems that provide clarity at each level of the organization about the standards for success and yet provide room for staff to be creative about how they achieve these standards.
- Managers acknowledge when staff members are not doing their work well…managers are not afraid to make tough personnel decisions so that the organization can live up to the promises it makes.
The Center for Employment Opportunities (CEO) is an example of how strong management is necessary to create a culture of high-performance. CEO employs people entering parole in New York State in transitional jobs at government facilities while helping them access better paying, unsubsidized employment. CEO Chief Operating Officer, Brad Dudding described to me how CEO management created, over the past 10 years, a culture and system of high performance.
Here is his story:
In the early years, CEO focused program performance on meeting individual contract milestones, not a set of unified organizational outcomes. They were proficient in collecting data and reporting it to funders, but did not use data to track participant progress, to make course corrections, and to manage to short-term outcomes.
In 2004 the Edna McConnell Clark Foundation provided CEO with a multi-year capital investment to:
- Create a theory of change as a blueprint for program intervention and outcomes measurement.
- Develop a performance measurement system to track progress toward those outcomes.
- Nurture a performance culture that uses data to understand program progress, build knowledge and correct performance gaps.
First, CEO management had to agree on a theory of change and the specific outcomes for which the organization would hold itself accountable. Next, management shared the theory of change with staff and demonstrated how each staff member contributed to its achievement through an all staff event, follow-up trainings and consistent messaging that the organization was entering an exciting period of change. CEO then adopted a new performance measurement system to reinforce the theory of change.
But reorienting the organization was not easy. Not everyone was ready to embrace a new culture of performance accountability and data tracking. CEO management was initially surprised by staff resistance and responded impatiently with compliance measures. Looking back, not enough time was invested in staff training and promoting the value proposition of new changes. At times it was an enormous effort to get front line staff to track and use data everyday to ensure participant goals were being met.
But the tipping point came when CEO promoted early adopters of the data system to management positions. These new managers were comfortable operating in a data-driven environment and holding others accountable to use data to track program participants’ progress. Once there was a group of strong managers in place, CEO’s performance culture started to take hold and program outcomes improved.
By 2010, CEO was managing to annual performance targets and short-term outcomes through staff’s real-time documentation and data analysis.
In 2012, the results of a three-year randomized control trial showed that CEO’s program resulted in a reduction in recidivism of 16-22%. But the evaluation also uncovered a need to improve CEO’s strategies for advancing long-term employment and for connecting individuals to the full-time labor market. In response, CEO created a job retention unit and developed innovative job retention strategies, including training programs and financial incentives for participants.
In 2013, CEO entered the New York State Social Impact Bond, the first state-sponsored transaction, through which CEO will serve 2,000 high-risk parolees in New York City and Rochester between 2014 and 2018. If CEO hits benchmarks and reduces the use of prison and jail beds by program participants, investors will be repaid their principal and will receive a return of up to 12.5% by the U.S. Department of Labor and New York state.
The tenets of a performance based culture — supportive leadership, disciplined managers, goal setting, data collection and analysis to track and improve outcomes — are now fully accepted by CEO staff and reinforced by management. CEO now has a highly developed system of tactical performance management, which allows the organization to know on a daily basis if it is delivering on its promise to its participants.
Photo Credit: Australian Paralympic Committee
In today’s Social Velocity interview I’m talking with Mary Kopczynski Winkler, senior research associate with the Center on Nonprofits and Philanthropy at the Urban Institute. Mary is a nationally recognized expert in the field of performance measurement and management. She is a founding member of the Leap of Reason Ambassadors Community, a private community of nonprofit thought leaders and practitioners committed to increasing the expectation and adoption of high performance in the social sector and who released the Performance Imperative earlier this year.
You can read past interviews in the Social Velocity interview series here.
Nell: PerformWell is an effort among Urban Institute, Child Trends and Social Solutions to offer tools and strategies for human services nonprofits to measure their work. How successful has this effort been and what are your plans for continuing to grow the capacity of nonprofits to measure their work?
Mary: PerformWell is a free, interactive, web-based resource designed to help human services nonprofits gain knowledge about performance management, access tools and resources they need to better service clients and meet outcomes, and obtain strategies for effective, efficient service delivery. Launched in March 2012, the demand for PerformWell has exceeded our expectations with more than 400,000 visitors (from all 50 states and more than 200 countries); 25,000 individuals have registered for our webinars; and more than 140,000 assessment tools have been downloaded from our site. Webinar survey results are routinely high, but we are working to put additional systems in place to track how nonprofits are using various aspects of PerformWell and to what end.
In 2013, the PeformWell partners engaged in a business planning process with Root Cause. Market research confirmed our views about a large unmet need for performance measurement knowledge and high interest in the resources offered through PerformWell, but that additional products and services are also desired, such as webinar training series, regional user conferences, and customized engagements with nonprofits. Users wanted a more interactive web-experience.
Our short- to medium-term goals include substantial updates to the website to improve the user experience (we also plan to solicit user feedback during and after these changes are implemented); development of additional products and services better aligned with the feedback obtained from the market research undertaken by Root Cause; and exploration of partnerships and sponsorships with nonprofits, consultants and funders to generate additional revenue and resources to expand the content, reach and use of PerformWell to improve the adoption and application of performance measurement and management practice across the nonprofit sector.
Nell: Some believe that measurement is perhaps more straightforward for human services nonprofits — you can measure change to an individual’s behavior or life circumstances — but measurement is more difficult for arts organizations or advocacy groups. What are your thoughts on that?
Mary: Sometimes I think this argument serves as a convenient excuse for organizations to avoid putting even the most basic systems in place to track progress or otherwise hold themselves accountable to their constituents. In 2007, with support from the Hewlett Foundation, the Urban Institute and the Center for What Works, we published a series of simple frameworks, as part of our Outcome Indicators Project, to help nonprofits in 14 program areas engage in performance measurement. Two of these areas are advocacy and performing arts. The Urban Institute also provided research support to the Performing Arts Research Coalition (PARC) to develop standardized surveys to help performing arts organizations across the country obtain more routine and better data from audience members, subscribers, and the community.
Establishing a causal link between advocacy or arts interventions and impact is, in my view, more challenging than for human service organizations. In the case of advocacy organizations, it can be very difficult to isolate the contributions of a particular campaign or even organization to a policy or legislative outcome.
It is, however, possible to devise strategies for capturing information on earlier stage outcomes, such as increased awareness.
I recently participated on a panel at the annual OPERA America conference – on “internal metrics for civic impact.” As much as measurement activities have evolved from the days of the PARC coalition, I observed that most of the metrics and data points were still very internally focused on measures of participation and attendance and fall well-short of anything approximating community or civic impact. I encouraged those present to consider stepping away from a focus on the impact of an individual opera company’s contribution to civic impact, and recommended instead more of a collective impact approach in collaboration with other arts, civic, and education organizations in a community.
In this case, I even hesitated to use the word “impact,” and suggested the group consider distinguishing between collective contribution toward a modest set of civic outcomes (e.g., performing arts promote understanding of other cultures or are a source of pride for those in the community) and the more traditional causal attribution usually reserved for the term “impact.”
Nell: Caroline Fiennes, among others, has argued that individual nonprofits should actually do less evaluation and rather rely on larger research studies to prove their theories of change. What do you make of that argument and the difference between evaluation and measurement?
Mary: I agree with some of what Caroline puts forth here – particularly her observations about “withholding (unflattering research) and publication bias” – an issue that University of Wisconsin-Madison professor Donald Moynihan has termed “performance perversity.” I also agree both with her suggestion that evaluations be done by a third-party to reduce any tendencies toward subjective reporting or bias and her endorsement of a greater consideration of shared metrics.
I am troubled, however, by the fact that only 7% of UK social-purpose organizations are interested in improving services, and her somewhat cavalier suggestion that monitoring and evaluation “wastes time and money.” Although she is not alone in this second argument (see for example Bill Shambra’s “take-down” of Charity Navigator’s efforts to encourage greater use of performance metrics in “Charity Navigator 3.0: The Empirical Empire’s Death Star?”), such sweeping generalizations undermine the legitimate and courageous attempts of many nonprofits to use data for program improvement efforts.
I agree with Phil Buchanan in that there is a “moral imperative” to make an honest attempt to understand if resources are being used effectively and certainly to guard against the possibility that programs could be doing more harm than good as organizations like Latin American Youth Center and Harlem Children’s Zone have discovered and since corrected.
I see measurement as a necessary practice for every nonprofit. But measurement is different from evaluation. Nonprofits need to start by developing a measurement infrastructure that makes sense for their organization – one that supports their mission and commitment to serve and improve the lives of their clients or constituents – not one that is reactionary and responsive to funders. It is precisely this kind of infrastructure that can lay the groundwork for a more rigorous evaluation, at a time that is right and appropriate for the organization’s stage in development.
I see measurement and evaluation along a continuum of inquiry that should be designed to support the learning objectives of an organization. Measurement helps organizations to take the day-to-day or month-to-month pulse of various activities and program results – these snapshots in time or scorecards help managers and service providers understand trends and provide an opportunity to correct, modify or otherwise adapt operations.
Evaluation is, by definition, more rigorous, more expensive, and takes considerably more time to see results. Evaluation serves a very important role as organizations make decisions about whether to continue, grow, scale or otherwise expand services, but it needs to occur at the right time – and certainly not as an organization is just getting off the ground.
Nell: It is difficult for most nonprofits to find funding for measurement work. For example, in the most recent Nonprofit Finance Fund State of the Sector survey, 69% of nonprofit respondents said their funders rarely or never cover the costs of measurement. How do we change that, or can we?
Mary: Although I am sympathetic to this argument and argue frequently that foundations have a unique and critical role to play in helping to build the capacity of nonprofits to better engage in measurement and evaluation, I think we need to change the conversation to one that focuses on the shared responsibility between nonprofits and funders for making the necessary investments in measurement and evaluation.
If nonprofits are truly ready to embrace a culture of measurement and high performance, then they need to reorganize operations in ways that embed measurement practice at every level of the organization, and change expectations from front-line workers all the way to the board of directors.
This means things like: defining expectations about data collection in job descriptions; setting aside a small percentage of funding for evaluation as a line-item in every grant request; and using data in meaningful ways in everyday discourse. Likewise, funders need to work more collaboratively with grantees to understand the data needs and capacity of nonprofits, consider funding longer-term grants that build in support for measurement and evaluation, and stop asking for data or reports that aren’t part of the conversation about continuous improvement and learning. Funders, too, can support field-building efforts to develop additional tools and resources in support of the measurement work nonprofits seek to accomplish.
There are a number of exemplary efforts already underway including Edna McConnell Clark Foundation’s Propel Next and the World Bank Group’s support of Measure4Change and the East of the River Initiative. Each of these efforts feature: targeted grants to build measurement and evaluation capacity of participating nonprofits; access to technical assistance resources; and a community of practice to help grantees learn from each other, share successes or failures, and reduce what is all too often a sense of isolation among measurement and evaluation practitioners.
Photo Credit: Urban Institute
Over the past few years I’ve developed a Social Velocity library of books, step-by-step guides, and webinars. My hope is that these tools can make the concepts I use with my consulting clients accessible to smaller and start up nonprofits who aren’t ready for or interested in a customized approach.
The tools follow the methods I develop in my consulting practice (like creating a financing plan, growing the board of directors, designing a theory of change) so when my consulting approach changes over time, the tools must change as well.
Which brings me to the Design a Theory of Change Guide. I created this guide a couple of years ago, but I recently changed the Theory of Change framework I use with my clients. I used to follow a more traditional logic model approach, but over time I’ve come to realize that there are really five specific and complex questions that make up a Theory of Change.
And those are:
- What is the target population or populations you are seeking to benefit or influence?
- What relevant trends in or changes to the external environment are occurring?
- How and where are your core competencies employed?
- What changed conditions do you believe will result from your activities?
- What evidence do you have that this theory will actually result in change?
The completely revised Design a Theory of Change Guide walks you step-by-step through answering these questions and creating your nonprofit’s own Theory of Change.
A Theory of Change is a fundamental building block to everything that your nonprofit does. Because without a Theory of Change, you won’t know what you are trying to accomplish, how you will get there, or whether you are moving towards it, and you certainly won’t attract the funding necessary to get there.
A Theory of Change can strengthen your nonprofit in many ways:
- Guides your strategic planning process. If you understand your nonprofit’s overall Theory of Change and what you exist to do, it is much easier to chart a future course.
- Helps revise the vision and mission of your organization, making them stronger and more compelling.
- Gives a framework to prove whether you are actually achieving results and creating real social change.
- Provides a filter for new opportunities as they arise. Do new opportunities fit within your Theory of Change?
- Engages board members and other volunteers, friends and supporters in your work. If people understand the bigger picture, they will be more inclined to give more time, energy, and other resources to the work.
- Allows staff to understand how their individual roles and responsibilities fit into the larger vision of the organization. This can increase staff morale, productivity, communication and overall commitment to the organization.
- Provides the basic argument for a case for investment or other fundraising messaging. With a Theory of Change, you can articulate what you are working to achieve, in a compelling way.
A Theory of Change is so fundamental because you cannot chart a strategic direction if you don’t know what you are trying to change. And you can’t prove that you’ve changed something unless you have articulated what it is that you want to change in the first place. And you certainly can’t convince funders, volunteers, and key decision makers to support you if you can’t tell them what you are trying to change and whether you are actually doing it.
So to truly create long-term social change you must start with a Theory of Change, which is why I encourage every nonprofit engaged in social change to create one.
You can learn more about the Design a Theory of Change Guide and download a copy of it. If you downloaded the previous Theory of Change Guide and would like the newly revised version free of charge, let me know, and we’ll send it to you.
As always, you can see all of the Social Velocity books, guides and webinars available for download on the Social Velocity Tools page.
April was another busy month in the world of social change writing. From Google’s shift to mobile, to the Baltimore protests, to using sitcoms to change public opinion, to the pace of social change, to teens and social media, to a new way to measure a country’s performance, there was much to read and digest.
Below are my 10 picks of the best in the world of social change in April, but please add to the list in the comments. And to see what else I found beyond these 10, follow me on Twitter, Facebook, Google+, or LinkedIn.
And you can read past months’ 10 Great Reads lists here.
- There was much analysis about what went wrong in Baltimore, but I found the most insightful to be Dan Diamond’s Forbes piece about how it is fundamentally a “tale of two cities” and the persistent inequality between two very different Baltimores.
- As is Google’s way, they made a huge change to their search algorithm in late April that will affect us all. Google is now favoring websites that are mobile friendly. But fear not, Beth Kanter offers some advice for upgrading your nonprofit’s website.
- For those in the trenches, the pace of social change can seem glacial. But this great graphic from Bloomberg demonstrates that for many issues (prohibition, interracial marriage, women’s suffrage, same-sex marriage) there was a tipping point at which America very quickly changed its mind. Fascinating.
- Civic Tech, or using technology to make citizens more engaged and government more effective, is a huge investment opportunity, says Stacy Donohue from the Omidyar Network. With venture capitalists, the federal government and nonprofit and for-profit solutions all poised to make change, Donohue sees civic tech as a “very real, very now investment opportunity.” Let’s hope that new ideas and (most importantly) lots of new money can turn our struggling democracy around.
- Social change can happen in many different ways, including by altering popular culture. Former Daily Show correspondent Aasif Mandvi is attempting this kind of shift with his new web sitcom that takes a “Cosby Show” approach to portraying American Muslims in order to combat Islamophobia.
- Writing in Slate, Krista Langlois takes a hard look at her fellow environmental journalists and whether they have failed to adequately describe the environmental challenges facing our planet since American concern about climate change has actually declined in the last 20 years.
- One of the most common hurdles to nonprofits raising capacity dollars is the challenge of articulating to funders the potential impact of a capacity investment. Grantmakers for Effective Organizations (GEO) have put together some tools to help funders understand the importance of and return on capacity investments. Share these with your funders.
- In April, MIT and the Social Progress Imperative launched the Social Progress Index, an effort to create a complement to the Gross Domestic Product that measures a nation’s social and environmental performance. The Social Progress Index looks at 52 indicators of a country’s social and environmental performance (like child mortality rate, adult literacy rate, greenhouse gas emissions). As Michael Porter, one of the chief architects behind it puts it, “Measuring social progress offers citizens and leaders a more complete picture of how their country is developing. And that will help societies make better choices, create stronger communities, and enable people to lead more fulfilling lives.”
- Writing on the Huffington Post Politics blog, Robert Reich describes a worrying trend where nonprofits are silencing themselves for fear of losing their big donors. As he writes, “Our democracy is directly threatened when the rich buy off politicians. But no less dangerous is the quieter and more insidious buy-off of institutions democracy depends on to research, investigate, expose, and mobilize action against what is occurring.”
- And finally, if you want to understand where social media is going, Pew Research Center released their most recent findings about teens use of social media and technology.
Photo Credit: Patrick Neil
As I mentioned last month, the Leap Ambassadors (of which I am a member) recently released the Performance Imperative, a detailed definition of a high-performing nonprofit. Because I think the Performance Imperative is so important and every nonprofit leader should understand it and begin to use it, today I am kicking off a series to describe, one-by-one, each of the seven pillars of the Performance Imperative.
I think the Performance Imperative is so exciting because it can serve as a north star to the nonprofit sector, helping organizations analyze their own performance and create a clear roadmap for improvement.
As Lowell Weiss, one of the leading architects of the Performance Imperative, explained in my interview with him last month:
High performance is all too rare in our sector today. In fact, we don’t even have a commonly accepted definition of the term “high performance.” The Performance Imperative is our attempt to create that common definition and then start the process of creating guideposts to help nonprofits who are motivated to improve their performance for the clients and causes they serve.
So, first up in this series on the Performance Imperative is Pillar #1: Courageous, Adaptive Executive and Board Leadership.
Without true leadership, at both the board and staff level, you will achieve little as a nonprofit. This pillar is about asking hard questions, pushing the organization toward excellence, continuously improving and taking nothing for granted.
You can read the full description of Pillar #1 in the Performance Imperative, but here are a few key elements present in nonprofits that exhibit this pillar:
- Boards “ask probing questions about whether the organization is living up to its promises and acknowledge when course correction is needed.”
- Executives and boards “know that great talent is a huge differentiator between organizations that are high performing and those that aren’t.”
- Executives and boards “know that they haven’t figured it all out and acknowledge that they still have a lot of work to do.”
- Executives and boards “are constantly assessing not only what the organization should be doing but also what it should stop doing…redirecting scarce resources to the highest opportunity areas.”
In other words, nonprofit leaders who embody Pillar 1 of the Performance Imperative, ask hard questions, build a stellar staff, seek continuous improvement, and put resources to their highest and best use.
There is no doubt that there are many examples of this courageous, adaptive leadership in the nonprofit sector. One of those, I believe, is Molly Baldwin, founder and CEO of Roca.
Molly founded Roca in 1988, and by 2004 it was a multi-million dollar teenage pregnancy and violence prevention program. But that year, Molly began asking some hard questions about the results Roca was achieving. She forced board and staff to take a huge step back and examine what they were doing and the ultimate effect that work had. She led her board and staff through a rigorous refocusing and pruning effort to limit their target populations and use data to drive their interventions. Instead of continuing a laundry list of services to many different populations that had limited effect, she helped her organization refocus resources on where they could create real change — transforming the lives of young men in the criminal justice system.
It was a challenging transition to lead, but the results are impressive. An internal study overseen by Harvard’s Kennedy School of Government in 2013 found that Roca reduced recidivism 65% and increased employment by 100% for the men in the program. And Roca was chosen as the lead provider in Masschusetts’ first pay for success effort.
Ten years ago Molly could have continued on Roca’s then current path, continuing to do “good work,” but failing to ask hard questions about whether that work was really resulting in change. But instead, Molly brought everything to a halt and forced board and staff to grapple with some fundamental and incredibly risky questions. In the end Molly’s leadership transformed Roca into an organization that is truly delivering solutions.
That’s the kind of social change leadership we need.
If you want to learn more, download the Performance Imperative and read additional case studies here.
Photo Credit: William B. T. Trego painting depicting George Washington’s army at Valley Forge.