In today’s Social Velocity interview, I’m talking with Isaac Castillo, Director of Outcomes, Assessment, and Learning at Venture Philanthropy Partners, where he leads VPP’s approach to data collection, data reporting, and outcome measurement.
Prior to coming to VPP, Isaac served as the Deputy Director for the DC Promise Neighborhood Initiative (DCPNI). At DCPNI, Isaac led efforts to improve outcomes in the Kenilworth-Parkside community in Ward 7 of the District of Columbia through the strategic coordination of programmatic solutions and research-based strategies. Prior to his time at DCPNI, Isaac served as a Senior Research Scientist at Child Trends where he worked with nonprofits throughout the United States on the development and modification of performance management systems and evaluation designs. In addition, Isaac was also the Director of Learning and Evaluation for the Latin American Youth Center (LAYC) where he led the organization’s evaluation and performance management work.
You can read interviews with other social change leaders here.
Nell: You have spent your career using data to improve the performance of the nonprofits for which you worked. Why do you think performance management is so important for nonprofits? Do you think all nonprofits should pursue performance management? When does it make sense and when doesn’t it?
Isaac: I believe that every nonprofit should pursue some form of performance management because they owe it to the clients they serve. Most nonprofits will assume that they are making a positive difference in people’s lives, but in the vast majority of cases they are just guessing. Using some form of performance management will allow every nonprofit organization to confirm this thinking and to identify areas that can and should be improved so that the next cohort of participants can get better services than the last.
Unfortunately, one of the greatest challenges preventing a nonprofit from implementing some form of performance management isn’t a lack of resources, expertise, or time. It is fear. The fear that they will find out that their work isn’t having a positive effect. This fear is what nonprofit leaders need to overcome, not for the benefit of themselves or their organization, but because they owe it to the clients they serve today and the clients they will serve in the future. I believe that every nonprofit should strive to serve tomorrow’s clients better than today’s clients, and one of the only ways to ensure that this happens is the sustained use of performance management.
The type of performance management that each nonprofit should pursue should vary by the size and scope of their work. At a minimum, small nonprofits should be tracking basic demographic and attendance information on their participants, and hopefully at least one meaningful output or outcome. Whether this occurs in a computerized system or in a spiral paper notebook is up to the nonprofit. But it doesn’t have to be costly, and it doesn’t take expertise. It only takes the will and desire to improve as a nonprofit.
Nell: In the nonprofits in which you’ve worked how have you been able to secure resources to fund performance management? What is the case you and your colleagues have made to funders and what do you think it will take to get more funders investing in performance management?
Isaac: Raising funding for performance management work usually takes a mix of several different strategies and approaches for potential and existing funders.
First, I strongly encourage nonprofits to include some percentage (1 to 5 percent – possibly more) of funding in each grant submission or proposal dedicated to supporting performance management and outcome measurement work. By placing this small percentage into each proposal, a nonprofit can begin to raise funds for internal evaluation and performance management activities. It may not seem like a lot, but it can add up, and eventually generate enough funds for a half-time or full-time position to support in-house performance management work.
Second, I also strongly encourage nonprofits to engage in regular ‘funder education’ – where a nonprofit proactively meets with their funders to have ongoing conversations about outcome measurement and evaluation. This allows both the funder and the nonprofit to come to agreement on measurement expectations and to ensure that both groups are focused on the same concepts. I often suggest that the first of these types of meetings focuses on each group’s definitions of three commonly misunderstood terms: outputs, outcomes, and impact.
Finally, I would recommend that the nonprofit and funder have an honest discussion regarding expectations of results and the funding necessary to support the related evaluation work. If a funder is expecting an random control trial (RCT) to be completed to determine ‘impact,’ then the nonprofit should be willing to push the funder to support a large investment to pay for a high quality evaluation. If the funder is only willing to support a small amount for outcome measurement, then the nonprofit should clearly articulate what is possible.
Nell: Ken Berger and Caroline Fiennes recently argued that we may have gone too far by asking nonprofits to produce research about their own outcomes. What’s your response to that argument?
Isaac: I fully support Ken and Caroline in their argument that most nonprofits should stay away from trying to produce impact research. The desire for ‘impact’ is something that has been (and continues to be) pushed unfairly (and without financial support) by the funding community.
I honestly think a lot of confusion in this space comes from inconsistent use and understanding of the term ‘impact’. The term ‘impact’ has a precise definition among researchers but is often used in a much broader context among funders, nonprofits, and the general public. In the research and evaluation world, impact is used to describe the effectiveness of a program while eliminating as many potential confounding factors as possible. That is why the use of random control trials (RCTs) is usually the cornerstone of impact research – RCTs are the easiest way to control for and eliminate confounding factors.
When most non-researchers use the term ‘impact’ however, they are usually just asking if the program or organization works and if it is making a difference for its intended service population. That is a much lower bar to set, and yet it is a critical distinction in this discussion. If you are thinking about ‘impact’ as a researcher, you will need a large amount of resources and expertise to determine ‘impact,’ which usually means completing one or more formal evaluations. If you are thinking about ‘impact’ in the more general sense and less strict way, then pursuing some form of performance management system will allow a nonprofit to determine if their efforts have been successful.
I do think every nonprofit should pursue some form of performance management to ensure that their work is having a positive effect as a complement to existing research that others have done. Relying only on the use of others’ research does not guarantee that a nonprofit will provide effective services and achieve positive outcomes. This type of research is a like a recipe – it shows what has worked in the past and provides a guide for the nonprofit – but a recipe can still be ruined with poor implementation or planning.
Every nonprofit has an obligation to the people they serve (and not to their funders) to ensure that their programming is having a positive effect (or at the very least not causing harm). Without some form of performance management system in place (even one that just uses paper and pencil), a nonprofit will never know if they have strayed too far from the recipe provided by previous research.
I also think there are a growing number of very sophisticated nonprofits that should be using AND producing research on effective programs. Every year, I see more and more nonprofits that hire talented and unbiased researchers dedicated to internal evaluation and outcome measurement work. These individuals are just as talented and unbiased as their colleagues working in traditional research and evaluation organizations. They can, and should, produce original research that can help inform the nonprofit field. The real challenge comes in nonprofit organizations finding the resources to support the hiring and retention of these individuals. Not every nonprofit will have the resources or capacity to hire one or more of these individuals – but those that do should absolutely be trying to produce original outcome and impact research to provide ‘recipes’ for effective programming that nonprofits with fewer resources can use in the future.
Nell: Your former organization, DC Promise Neighborhoods, is part of the national Promise Neighborhoods Initiative launched by the US Department of Education in 2010 and modeled after the famous Harlem Children’s Zone. How successful has this national replication of a successful local model been? Have you been able to replicate outcomes? And what hurdles, if any, have you and other replication sites found?
Isaac: I think that there has been some initial success among the Promise Neighborhoods. Part of the challenge that all the Promise Neighborhoods face is that the Harlem Children’s Zone did not achieve their success overnight. They have been working in Harlem for decades, so it would be unrealistic to believe that the Promise Neighborhoods would be able to create large scale change in a matter of a few years.
However, there are signs of progress across all of the Promise Neighborhoods. Each of the Promise Neighborhoods started to address a few outcomes with the initial round of funding, and these outcomes varied. Some focused on math and reading proficiency for students, some focused on obtaining medical homes for young children, and others sought to increase the amount of healthy food consumed by residents. In DC, we focused on improving school attendance.
I do think that most of the 12 Promise Neighborhood Implementation grantees were able to make progress on the outcomes they identified as initial focus areas. However, the very nature of the work (creating community level change) doesn’t lend itself to the rapid accomplishment of multiple outcomes in a short period of time. Each of the Promise Neighborhoods had to prioritize certain outcomes for their respective communities, and only several years later are they able to claim success and begin to identify the next set of outcomes to be addressed. So while certain outcomes haven’t necessarily been replicated across all the Promise Neighborhoods, that is due to the differences in priorities and community conditions rather than any problem with the model itself.
Photo Credit: Venture Philanthropy Partners
Note: I was asked by The Center for Effective Philanthropy to review their latest research report, Sharing What Matters: Perspectives on Foundation Transparency, released in late February, and provide my thoughts about it for their on-going blog series on the report. Below is my post which originally appeared on the CEP blog.
Sharing What Matters: Perspectives on Foundation Transparency provides some startling data about the state of transparency in the foundation world.
While for the most part, foundation leaders recognize the importance of transparency and are trying to be more transparent, the report shows there is still much work to do.
To me, this question of foundation transparency is part of the larger, ever-present power imbalance in the nonprofit sector between those with money (funders), and those who seek that money (nonprofits). Funders often encourage nonprofits to be transparent about their results and when they have succeeded or failed. But it appears that in these two areas (results and lessons learned), funders are less transparent than either their grantees want them to be, or they would like themselves to be.
This is all critically important because a more transparent philanthropic sector — particularly if foundations were more transparent about how they assess their results and what has worked and what hasn’t — could mean more money flowing to more social change.
CEP’s report delineates two levels of foundation transparency. First is transparency about grantmaking: who leads the foundation, how they have made grants in the past, how they make decisions. The second is transparency about the results foundations themselves achieve: how they assess the performance of their investments, how they share successes and failures.
This second (and I would argue much more interesting) level of transparency is about foundations reporting the very thing they are often asking nonprofits to report: their performance.
In particular, the research uncovers three stark disconnects:
- Foundations Don’t Share How They Assess Their Performance
Of the foundation leaders surveyed, 61 percent said they believe being transparent about how their foundation assesses its performance could increase effectiveness to a significant extent. Yet, only 35 percent of foundations reported actually being very or extremely transparent about it.
- Foundations Aren’t Transparent about Successes and Failures
While 69 percent of foundation leaders think that being transparent about what’s worked in their grantmaking could increase their effectiveness, only 46 percent report being very or extremely transparent about what’s worked. And transparency about what hasn’t worked is even worse. 30 percent of foundation leaders say their foundations are very or extremely transparent about what does not work, which makes failures the lowest-rated area of foundation transparency. And nonprofits agree that foundation transparency is lowest when it comes to sharing what hasn’t worked.
- Foundations Want to Be More Transparent, But Aren’t
While 94 percent of foundation leaders surveyed say that increased transparency is a medium or high priority at their foundation, 75 percent of foundation leaders say that their current levels of transparency are not sufficient. And shockingly, 24 percent of foundation leaders say that nothing limits their ability to be more transparent. So it’s a big priority, yet it’s not getting done.
The report suggests some reasons why transparency about performance and lessons learned is recognized as important, but still far from ubiquitous in the philanthropic sector:
- Lack of Strategy: Foundations aren’t creating clear enough goals around which they can actually assess their performance.
- Lack of Capacity for Evaluation: Foundations aren’t allocating enough resources to assessing their performance.
- Fear of Diminished Reputation: Foundations are afraid of harming their own or their grantees’ reputations by revealing what has or hasn’t worked.
Surprisingly (or maybe not so surprisingly), these impediments to foundation transparency mimic the hurdles nonprofits find (or place) in their own way. Nonprofits often pour as much money as possible into programs and skimp on investing in organization-building efforts like strategy and evaluation. This bias against organization-building is often encouraged (or demanded) by their funders. And so it appears that funders put these same hurdles in their own way. Perhaps foundations, just like their nonprofit grantees, need to acknowledge that with sufficient investments in smart strategy and performance evaluation, greater results can be achieved.
The third and final impediment to foundation transparency about performance and lessons learned is trickier. Fear of harming the reputations of their grantees by sharing lessons learned is a real issue. Foundations tend to invest in packs. So if a foundation reveals investments that have failed, there is a risk that other foundations will flee.
But if we truly want to move to a place where more resources flow to what works, don’t we have to be more transparent about what worked and what didn’t work? If a foundation investment failed because of the foundation’s shortcomings (the investment didn’t fit with foundation goals, the foundation didn’t invest enough, or it didn’t invest in capacity as well as programs), the foundation (and other foundations learning from these lessons) could learn to become more effective investors. And if the investment didn’t work simply because it was the wrong intervention, then isn’t it better to move investments to interventions that do work? Fear can be a debilitating thing, and for the sake of greater results, I think both foundations and their nonprofit grantees must work to overcome it.
Ultimately, the CEP report is hopeful. It uncovers a desire among both foundation leaders and their grantees to move from a basic level of transparency toward a deeper (and more important) one that reveals performance and lessons learned.
Let’s hope that this stated desire for a change in foundation transparency, and the requisite changes in how foundations invest in strategy and performance assessment and overcome fear, becomes reality.
Photo Credit: The Center for Effective Philanthropy
From an historic blizzard that blanketed the country, to tackling poverty, to the leadership of Black Lives Matter, to technology in the new year, to using social media to stop ISIS, to advice for Charity Navigator, January was an interesting month in the world of social change.
- Winter storm Jonas dumped several feet of snow across the country, but also offered a couple of interesting lessons in social change. First, the sheer amount of snow piled up on east coast urban streets provided a glimpse into better urban design. And after the blizzard hit Washington, DC it seems only female senators were brave enough to come to work. Among them, Senator Lisa Murkowski wondered: “Perhaps it speaks to the hardiness of women…that put on your boots and put your hat on and get out and slog through the mess that’s out there.”
- Writing in the Nonprofit Quarterly, Tom Klaus took issue with those who criticize the Ferguson and Black Lives Matters movements as being “leaderless.” Instead, he argued that they demonstrate a more effective “shared leadership” model: “Shared leadership…means that multiple members of a team or group step up to the responsibility and task of leadership, often as an adaptive response to changing circumstances. Multiple members may emerge to lead at the same time, or it may be serial as multiple leaders emerge over the life of a team or group.” And The Chronicle of Philanthropy profiled three of the leaders of the Black Lives Matter movement.
- One of my favorite bloggers, David Henderson, has made a new year’s resolution to write more often. Let’s hope he keeps it up because he offered us two great ones this month. First, he wrote a scathing critique of the nonprofit and philanthropy sectors for not standing up against presidential candidate Donald Trump’s hate-filled ideology. And then he took it further in a later post arguing that the philanthropic sector must get more political: “It seems a strange consensus that philanthropy and politics do not mix. Yet it is our politics, and more specifically our collective values, that creates the maladies we aim to address. Martin Luther King was a civil rights pioneer not for creating a nonprofit that provided social services to help African Americans live a little better, but by challenging the laws and social values that subjugated a significant portion of our community. Social interventions like homeless shelters, food pantries, and tutoring programs are fundamentally responses to injustice. While these programs are wrapped in apolitical blankets, they are plainly and intuitively critiques of the system we live in.”
- And speaking of critiques, columnist Tom Watson wrote a sharp commentary on American philanthropy arguing that it is going the way of American politics — moving from democracy towards plutocracy: “The disparity between democratic philanthropy and its plutocratic cousin is nowhere more apparent than in the importance placed on the Facebook co-founder’s commitment to giving away much of his vast personal fortune compared with the potential of the largest digital social network in the nation. Mr. Zuckerberg’s billions may create major causes and eventually steer public policy, but many nonprofits will struggle to find in their budgets the money required to purchase desperately needed social-media eyeballs from his advertising department. If there’s a better example of the power gulf in American philanthropy, I’m not sure what it is.”
- And other critiques of philanthropy in January went even further, with some arguing that modern American philanthropy attempting to address growing wealth inequality (illustrated by a new Oxfam infographic “An Economy for the 1%“) is a paradox because philanthropy itself emerged from the wealth excesses of capitalism. A new book by Erica Kohl-Arenas argued that philanthropic interventions to solve poverty have been flawed because they don’t address the structural issues causing the poverty in the first place. And her argument was extended when she wrote about her view of a January 7th public event at the Ford Foundation where Darren Walker (who recently announced a new foundation focus on overcoming poverty) and Rob Reich discussed these issues.
- Caroline Fiennes argued that nonprofits should not try to “prove their impact,” since proof of impact is impossible, but rather use evaluation to gain knowledge that can help “maximize our chances of making a significant impact.” Patrick Lester, writing in the Stanford Social Innovation Review, offered a similar caution about outcomes, but this time to the Obama administration: “A dose of…realism, combined with a greater reliance on evidence and a willingness to learn from the past, could transform the administration’s focus on outcomes into an important step forward. By openly acknowledging the challenges and dangers, recognizing the difference between mere outcomes and true impact, and demonstrating how this time we will do better, the administration could show that what it’s really calling for is not just an outcomes mindset, but an Outcomes Mindset 2.0.”
- Speaking of proving results, Charity Navigator’s new leader, former Microsoft exec Michael Thatcher, and the board that hired him came under attack in January for not moving quickly enough away from rating nonprofits on financials and towards rating them based on results. But Doug White, writing an opinion piece in The Chronicle of Philanthropy and who created the beginning data behind Charity Navigator many years ago, took it even further took it even further: “Charity Navigator is far worse than nothing. The best that could happen is for the group to sink into oblivion, with no charities, no news outlets, and no donors giving it any thought. Or the group could take serious steps to grow up, humbly taking the time and effort to truly try to understand the charitable world.”
- Wanting to get further into the social change game, Facebook COO Sheryl Sandberg announced a new effort to use Facebook “Likes” to stop ISIS recruitment efforts on social media. It will be interesting to see how effective this slacktivism effort becomes at creating real change.
- Kivi Leroux Miller released her annual Nonprofit Communication Trends Report, including lots of data about how and where nonprofits are marketing. And while she found that YouTube is currently the #3 social network for nonprofits, that may change since YouTube just announced new “donation cards” that allow donors to give while watching a video.
- And finally, in January we lost David Bowie. But Callie Oettinger urged us not to be sad, but rather, inspired: “I [am] comforted in thinking of Bowie…on Mars, mixing it up with other artists…a place where the greats go to keep an eye on the rest of us and send down jolts of inspiration from above.” Yes.
Photo Credit: Northside777
There is an interesting report out today on the effectiveness of the Social Innovation Fund (SIF). Authored by the Social Innovation Research Center (SIRC), a nonpartisan nonprofit research organization, the new report details what has worked and what hasn’t in the six year history of the SIF.
Launched by the Obama administration in 2009, the SIF — a program within the Corporation for National and Community Service — provides significant funding to foundations that follow a venture philanthropy model by regranting that growth capital, along with technical assistance, to evidence-based nonprofits in “youth development, economic opportunity, and healthy futures” areas. In 2014, SIF expanded its efforts to include a portfolio of Pay for Success (social impact bond) grantees.
Now, 6 years on it is interesting to take a look back to understand what, if any, effect SIF has had on the nonprofit sector. The effect of the SIF is also critical given that, as of right now, the House and Senate have both defunded SIF in their respective funding bills.
To date, the SIF portfolio is made up of $241 million of federal investments and $516 million in private matching funds, which was invested in 35 intermediary grantees and 189 subgrantee nonprofits working in 37 states and D.C.
The SIRC report focuses on the current progress of SIF grants made during the first three years of the program (2010-2012). The report finds two clear positive results for the SIF so far. The SIF has:
- Added to the nonprofit sector’s evidence base about which programs work, and
- Built the capacity of nonprofit subgrantees, especially in the areas of “performance management systems, evaluations, financial management, regulatory compliance systems, and experience with replicating evidence-based models.”
On the negative side, however, the report finds that the SIF put real burdens on funders and nonprofits with its fundraising match requirements and the federal regulatory requirements. The report also finds that the SIF has had little effect on the sector as a whole because the SIF has not very broadly communicated their learnings so far.
To me, of course, most interesting are the report’s finding about capacity building at nonprofit subgrantees. There is such a need for nonprofit capacity building in the sector, and this was a clear goal of the SIF.
The SIF is one of few funders that do more than pay lip service to performance management by actually investing in building the capacity of nonprofits to do it. However, the SIF has been criticized for mostly selecting nonprofits that already had strong capacity. And indeed, the SIRC report finds that the SIF was most successful among those nonprofits that already had high capacity (in performance management, fundraising function, etc.) prior to SIF funding. Indeed, the report found that “poorly-resourced intermediaries working with less well-resourced community based organizations have been at a disadvantage.”
One SIF grantee in particular, The Foundation for a Healthy Kentucky, really struggled to build the capacity of their subgrantees whose starting capacity was so low. As they put it:
During the course of participation, it became clear that…[SIF] was really better suited for replicating existing programs or, at a minimum, investing in well-established programs that had some level of sophistication around organization systems and evaluation.
This mirrors earlier criticism of the SIF that it was set up to grow only those nonprofits that were already doing well, while those nonprofits that struggled with basic capacity issues were left out. The SIF has struggled to determine whether it is funding innovation (new solutions with limited capacity), or proven solutions (with a long track record and the corresponding capacity). It seems the two are mutually exclusive.
What the SIF is trying to do is such tricky business. To identify, fund and and scale solutions that work is really the holy grail in the social change sector. Certainly there are hurdles and missteps, but I think it’s exciting when government gets in the social change game in a big way. Six years is really too soon to tell. So I hope that this brief SIF experiment is allowed to continue, and we can see what a social change public/private partnership of this scale can really do.
To read the full SIRC report go here.
Photo Credit: Obama signs the Serve America Act in 2009, Corporation for National and Community Service
In today’s Social Velocity interview I’m talking with Mary Kopczynski Winkler, senior research associate with the Center on Nonprofits and Philanthropy at the Urban Institute. Mary is a nationally recognized expert in the field of performance measurement and management. She is a founding member of the Leap of Reason Ambassadors Community, a private community of nonprofit thought leaders and practitioners committed to increasing the expectation and adoption of high performance in the social sector and who released the Performance Imperative earlier this year.
You can read past interviews in the Social Velocity interview series here.
Nell: PerformWell is an effort among Urban Institute, Child Trends and Social Solutions to offer tools and strategies for human services nonprofits to measure their work. How successful has this effort been and what are your plans for continuing to grow the capacity of nonprofits to measure their work?
Mary: PerformWell is a free, interactive, web-based resource designed to help human services nonprofits gain knowledge about performance management, access tools and resources they need to better service clients and meet outcomes, and obtain strategies for effective, efficient service delivery. Launched in March 2012, the demand for PerformWell has exceeded our expectations with more than 400,000 visitors (from all 50 states and more than 200 countries); 25,000 individuals have registered for our webinars; and more than 140,000 assessment tools have been downloaded from our site. Webinar survey results are routinely high, but we are working to put additional systems in place to track how nonprofits are using various aspects of PerformWell and to what end.
In 2013, the PeformWell partners engaged in a business planning process with Root Cause. Market research confirmed our views about a large unmet need for performance measurement knowledge and high interest in the resources offered through PerformWell, but that additional products and services are also desired, such as webinar training series, regional user conferences, and customized engagements with nonprofits. Users wanted a more interactive web-experience.
Our short- to medium-term goals include substantial updates to the website to improve the user experience (we also plan to solicit user feedback during and after these changes are implemented); development of additional products and services better aligned with the feedback obtained from the market research undertaken by Root Cause; and exploration of partnerships and sponsorships with nonprofits, consultants and funders to generate additional revenue and resources to expand the content, reach and use of PerformWell to improve the adoption and application of performance measurement and management practice across the nonprofit sector.
Nell: Some believe that measurement is perhaps more straightforward for human services nonprofits — you can measure change to an individual’s behavior or life circumstances — but measurement is more difficult for arts organizations or advocacy groups. What are your thoughts on that?
Mary: Sometimes I think this argument serves as a convenient excuse for organizations to avoid putting even the most basic systems in place to track progress or otherwise hold themselves accountable to their constituents. In 2007, with support from the Hewlett Foundation, the Urban Institute and the Center for What Works, we published a series of simple frameworks, as part of our Outcome Indicators Project, to help nonprofits in 14 program areas engage in performance measurement. Two of these areas are advocacy and performing arts. The Urban Institute also provided research support to the Performing Arts Research Coalition (PARC) to develop standardized surveys to help performing arts organizations across the country obtain more routine and better data from audience members, subscribers, and the community.
Establishing a causal link between advocacy or arts interventions and impact is, in my view, more challenging than for human service organizations. In the case of advocacy organizations, it can be very difficult to isolate the contributions of a particular campaign or even organization to a policy or legislative outcome.
It is, however, possible to devise strategies for capturing information on earlier stage outcomes, such as increased awareness.
I recently participated on a panel at the annual OPERA America conference – on “internal metrics for civic impact.” As much as measurement activities have evolved from the days of the PARC coalition, I observed that most of the metrics and data points were still very internally focused on measures of participation and attendance and fall well-short of anything approximating community or civic impact. I encouraged those present to consider stepping away from a focus on the impact of an individual opera company’s contribution to civic impact, and recommended instead more of a collective impact approach in collaboration with other arts, civic, and education organizations in a community.
In this case, I even hesitated to use the word “impact,” and suggested the group consider distinguishing between collective contribution toward a modest set of civic outcomes (e.g., performing arts promote understanding of other cultures or are a source of pride for those in the community) and the more traditional causal attribution usually reserved for the term “impact.”
Nell: Caroline Fiennes, among others, has argued that individual nonprofits should actually do less evaluation and rather rely on larger research studies to prove their theories of change. What do you make of that argument and the difference between evaluation and measurement?
Mary: I agree with some of what Caroline puts forth here – particularly her observations about “withholding (unflattering research) and publication bias” – an issue that University of Wisconsin-Madison professor Donald Moynihan has termed “performance perversity.” I also agree both with her suggestion that evaluations be done by a third-party to reduce any tendencies toward subjective reporting or bias and her endorsement of a greater consideration of shared metrics.
I am troubled, however, by the fact that only 7% of UK social-purpose organizations are interested in improving services, and her somewhat cavalier suggestion that monitoring and evaluation “wastes time and money.” Although she is not alone in this second argument (see for example Bill Shambra’s “take-down” of Charity Navigator’s efforts to encourage greater use of performance metrics in “Charity Navigator 3.0: The Empirical Empire’s Death Star?”), such sweeping generalizations undermine the legitimate and courageous attempts of many nonprofits to use data for program improvement efforts.
I agree with Phil Buchanan in that there is a “moral imperative” to make an honest attempt to understand if resources are being used effectively and certainly to guard against the possibility that programs could be doing more harm than good as organizations like Latin American Youth Center and Harlem Children’s Zone have discovered and since corrected.
I see measurement as a necessary practice for every nonprofit. But measurement is different from evaluation. Nonprofits need to start by developing a measurement infrastructure that makes sense for their organization – one that supports their mission and commitment to serve and improve the lives of their clients or constituents – not one that is reactionary and responsive to funders. It is precisely this kind of infrastructure that can lay the groundwork for a more rigorous evaluation, at a time that is right and appropriate for the organization’s stage in development.
I see measurement and evaluation along a continuum of inquiry that should be designed to support the learning objectives of an organization. Measurement helps organizations to take the day-to-day or month-to-month pulse of various activities and program results – these snapshots in time or scorecards help managers and service providers understand trends and provide an opportunity to correct, modify or otherwise adapt operations.
Evaluation is, by definition, more rigorous, more expensive, and takes considerably more time to see results. Evaluation serves a very important role as organizations make decisions about whether to continue, grow, scale or otherwise expand services, but it needs to occur at the right time – and certainly not as an organization is just getting off the ground.
Nell: It is difficult for most nonprofits to find funding for measurement work. For example, in the most recent Nonprofit Finance Fund State of the Sector survey, 69% of nonprofit respondents said their funders rarely or never cover the costs of measurement. How do we change that, or can we?
Mary: Although I am sympathetic to this argument and argue frequently that foundations have a unique and critical role to play in helping to build the capacity of nonprofits to better engage in measurement and evaluation, I think we need to change the conversation to one that focuses on the shared responsibility between nonprofits and funders for making the necessary investments in measurement and evaluation.
If nonprofits are truly ready to embrace a culture of measurement and high performance, then they need to reorganize operations in ways that embed measurement practice at every level of the organization, and change expectations from front-line workers all the way to the board of directors.
This means things like: defining expectations about data collection in job descriptions; setting aside a small percentage of funding for evaluation as a line-item in every grant request; and using data in meaningful ways in everyday discourse. Likewise, funders need to work more collaboratively with grantees to understand the data needs and capacity of nonprofits, consider funding longer-term grants that build in support for measurement and evaluation, and stop asking for data or reports that aren’t part of the conversation about continuous improvement and learning. Funders, too, can support field-building efforts to develop additional tools and resources in support of the measurement work nonprofits seek to accomplish.
There are a number of exemplary efforts already underway including Edna McConnell Clark Foundation’s Propel Next and the World Bank Group’s support of Measure4Change and the East of the River Initiative. Each of these efforts feature: targeted grants to build measurement and evaluation capacity of participating nonprofits; access to technical assistance resources; and a community of practice to help grantees learn from each other, share successes or failures, and reduce what is all too often a sense of isolation among measurement and evaluation practitioners.
Photo Credit: Urban Institute
There were some pretty exciting things happening in the world of social innovation last month. From a new fund to make philanthropy more effective, to a new blog series written by funders making the case for investing in nonprofit leadership, to some ideas for making performance measurement more accessible to small nonprofits and arts and culture organizations, to some interesting partnerships between philanthropy and city government.
It all made for a great month of reads. Below is my pick of the 10 best reads in social innovation in September. As always, add what I missed to the comments. And if you want a longer list, follow me on Twitter, Facebook, Google+ or LinkedIn.
You can read past months’ 10 Great Social Innovation Reads lists here.
- The Fund for Shared Insight, a collaboration among seven major foundations, launched in September. The group plans to “pool financial and other resources to make grants to improve philanthropy…to encourage and incorporate feedback from the people we seek to help; understand the connection between feedback and better results; foster more openness between and among foundations and grantees; and share what we learn.” They plan to be very transparent with this entire experiment. I can’t wait to see what develops.
- Another development in the realm of improving philanthropy was the launch of the Stanford Social Innovation Review blog series where foundation leaders discuss why and how they have invested in nonprofit leadership development. As I mentioned earlier, Ira Hirschfield from the Haas Foundation kicked off the series, and Surina Khan from the Women’s Foundation of California was next up. To have such an open dialogue about nonprofit capacity investments, particularly around leadership development, is amazing. Let’s hope it encourages similar conversations outside the blogosphere.
- And the third piece from the world of philanthropic enlightenment, Daniel Stid of the Hewlett Foundation wrote a great post about ending the nonprofit starvation cycle. As he put it, “Effective leaders need to be willing to take the risk of saying something that a funder might not want to hear when their organization’s long run effectiveness is at stake. If they are not, then shame on them. Funders, for our part, should fund the full cost of the work we are asking our grantees to undertake in a way that leaves their overall organization and its finances whole; if we don’t, then shame on us.” Amen!
- There is further evidence that philanthropy as we know it is changing – a new report by The Economist takes a hard look at how Generations X and Y (those born between 1966 and 1994) are transforming philanthropy, particularly around “a strong desire to have a measurable, enduring impact.” This is exciting because if donors increasingly invest based on results, we can shift more money to social change. As the authors of the report put it, “The young generation of givers is focused on data, measurement and demonstrable results. More than any other generation, they want to check facts, know all the information ahead of time and ensure that they are well-informed at every stage of the process.”
- And there was lots to say about measuring performance this month. The Foundation Center and WINGS, a global network of 90 support organizations serving philanthropy in 35 countries, announced the creation of The Global Philanthropy Data Charter to gather and share philanthropy data for public benefit.
- Measuring impact is complex and costly, but Carly Pippin from Measuring Success, offers 4 steps for how small nonprofits can assess impact affordably.
- Measurement is particularly challenging in the arts and culture arena because, as Natasha Bloor of The Old Vic Theatre explains, “There is an understandable reticence within the cultural and creative industries when it comes to proving the social value of art. For many, the arts have an intrinsic worth that cannot be mapped or measured, with the primary benefit found in creative self-expression itself, rather than the longer-term effects experienced afterwards.” But she offers a new approach that they have found very effective.
- And for a completely free way to assess the social value of building low-cost housing, child-care centers, and health clinics there is the Social Impact Calculator, developed by the Low Income Investment Fund. They developed the tool to measure the effect of their own work and then decided to share it.
- Stephanie Jacobs of the Nonprofits Assistance Fund offers some tips to turn your board into the financial leaders they need to be.
- And finally, there were some interesting examples of partnerships between local government and philanthropy aimed at strengthening cities. Rona Jackson from Living Cities described 5 ways philanthropy and local government can work together. And the Kalamazoo Promise, a partnership between local philanthropists and city schools that pays tuition at a Michigan college for any student who graduates from a Kalamazoo school, shows these ideas in action.
Photo Credit: Valerie Everett
Between my own time away from social media in August, the general end of summer quiet, and of course, the glut of posts about the Ice Bucket challenge (of which I have already said my piece), my list of great reads in August is admittedly slim.
But there was some interesting debate, most notably about “strategic philanthropy” and about ratings agency Philanthropedia. Also, calls for more nonprofit leadership development and for nonprofit leaders to get out of their own way by taking the Overhead Pledge. Throw in a little Mark Twain, some sharing economy, and a dash of Millennial analysis and you have a pretty good month in the world of social change.
So below is my pick of the 10 best reads in the world of social innovation in August. For an expanded list you can follow me on Twitter, Facebook, LinkedIn or Google+. And you can see past months’ 10 Great Reads lists here.
- Leadership development is a woefully underfunded need in the nonprofit sector. Indeed from 1992-2011 only $3.5 billion of the nearly $287 billion dollars granted by foundations went to support leadership. In order to get more foundations investing in leadership development, Rusty Stahl offers case studies of 9 foundations who already do.
- In the summer issue of Stanford Social Innovation Review, the lead article “Strategic Philanthropy for a Complex World” caused quite a stir in the philanthropy world with many arguing that there is not much new there. In August, Alliance Magazine ran a series of editorials by philanthropy leaders as counterpoints. Most interesting among them was Avila Kilmurray’s, former director of the Community Foundation of Northern Ireland, response, in which she said “Can we not just recognize that when any funder sets her/himself the task of addressing complex issues…there needs to be provision for continuous consultation, practice, reflection and change?”
- An interesting article in the New York Times paints the Millennial generation as a very communal-minded one, where “the highest value isn’t self-promotion, but its opposite, empathy — an open-minded and -hearted connection to others.” From working, to eating, to shopping it seems Millennials bake social into everything they do. How will the world be different if that holds true as they age?
- Writing in Forbes, Tom Watson asks whether nonprofits should participate in GivingTuesday. As he puts it, “Is #GivingTuesday a well-meaning marketing promotion – or is it a real, organic movement for change?…[Does it] seek to increase U.S. giving from 2% of GDP (where it’s been stuck for two generations) to some higher point?” Amen to that!
- Rating nonprofit effectiveness is such a tricky challenge. Philanthropedia, one rating system that is driven by crowdsourced feedback from experts, comes under fire from the clean water space for being just “a popularity contest.” But others claim it’s an improvement over previous evaluations.
- Writing in the Chronicle of Philanthropy Nicole Wallace shows the value of sharing data by profiling Crisis Text Line, which gives other nonprofits, researchers and government agencies access to their data of 60,000 counseling sessions with teens in crisis to use in their own programs. It begs the question whether other social change data could be shared and how we make that easier to do.
- Sue Dorsey from Water for People was among a group of nonprofit leaders at the InsideNGO conference who took the Overhead Pledge in August, vowing to fully disclose the true costs of their nonprofits. And she encourages other nonprofit leaders to follow suit. This is exciting because it is not enough for funders to get over the overhead myth, nonprofit leaders must as well.
- I am always a sucker for connecting literature and/or history to social change, and even better both, so David Bonbright’s post about how Mark Twain would have viewed recent trends in business is fascinating. Bonbright argues that Twain wanted American business to fully integrate profit and community. And we are beginning to witness this trend again where companies are “embracing the full implications of what they are – what they mean for the environment, for communities, for the most marginalized people affected by their supply chains…[because] this is best way to remain competitive and successful over time.” Let’s hope!
- The new “sharing” economy is not all good, but not all bad either, as Daniel Ben-Horin argues that “there are enormous opportunities for the social sector to engage with the values-driven segment of the sharing economy.”
- Finally, some guidance on making your nonprofit email marketing more mobile friendly and your website better able to connect people to your cause. It’s all about responsive, engaging design.
Photo Credit: Seth Anderson
Note: Third in my list of guest bloggers this summer is David Henderson. David’s professional focus is on improving the way social sector organizations use information to address poverty. Here is his guest post:
I was recently turned down for a position at a startup-up big-data company focused on the philanthropic sector because I’m “too pessimistic”. This company initially sought me out since they don’t have any social sector expertise on staff, a likely requisite to make successful nonprofit software. Our courtship turned sour when I expressed my view that we have a lot more social sector initiatives than evidence that those interventions actually work.
My skepticism that social sector initiatives by and large work was wrongly misconstrued as pessimism that social progress is possible. Skepticism is a critical driver of intellectual curiosity. I spent a pretty penny on two degrees that essentially taught me how to critically assess the divide between rhetoric and results. Indeed, the null hypothesis in a statistical model assumes the intended effect is not present. I guess statisticians are just a bunch of pessimists.
Fundamentally, I believe the company I interviewed with was mirroring the widespread lack of intellectual curiosity that plagues the social sector and impedes real progress. Too many nonprofits are terrified of having their claims of social impact investigated, lest their effects are discovered to be more modest than claimed. And I don’t blame them. The funding community’s emphasis on investing in “what works” has resulted in a proliferation of noise as every nonprofit steadfastly argues their interventions cure everything. It’s no wonder evaluators are seen as Angels of Death.
I generally don’t favor taking cues from the for-profit world, but venture capital and angel investors’ practice of investing in people and teams over ideas is far more conducive to intellectual honesty in product (and social intervention) development. The basic premise of this investment strategy is that initial product ideas are generally wrong, but smart people will investigate, iterate, and innovate.
Compare that philosophy to the social sector, where the expectation is that nonprofits already have the answers, they just need money to scale them up. This assumption is largely incorrect, but by making funding contingent on the perception of effectiveness, the nonprofit sector is incentivized to not question the efficacy of its own work. In this model, continued funding depends on a lack of intellectual curiosity at best, and intellectual dishonesty at worst.
A better alternative is for nonprofits to embrace intellectual curiosity, and to be the first to question their own results. Under this model, nonprofits would invest in their capacity to intelligently probe the effectiveness of their own interventions, by staffing those with the capacity to sift through outcomes data and investing in the growing list of tools that are democratizing evaluation. Of course, this would require a shift in the funding community away from “investing in what works” to more humbly “investigating what works”.
A shift toward intellectual curiosity would create more space for the sector to solicit beneficiary feedback in the design of social interventions, as organizations would no longer be incentivized to defensively “prove” existing approaches work, and instead would be rewarded for proactively evolving practices to achieve better results. It is this very intellectual curiosity that led organizations like GiveDirectly and the Family Independence Initiative to invest in the poor directly, a departure from long-standing anti-poverty practices that the evidence suggests might actually work. It’s a shame that organizations imbued with a mission of experimentation deviate so far from the norm.
I don’t consider it pessimistic to question whether the sector is achieving its intended social impact. To the contrary, it’s rather cynical to set aside what should be the critical question for any nonprofit organization in the name of self-preservation. In order to achieve social progress, the sector needs to expel anti-intellectual policies and actors in favor of a healthy skepticism that questions everything, and is willing to try anything.
Photo Credit: NASA