In today’s Social Velocity interview, I’m talking with Isaac Castillo, Director of Outcomes, Assessment, and Learning at Venture Philanthropy Partners, where he leads VPP’s approach to data collection, data reporting, and outcome measurement.
Prior to coming to VPP, Isaac served as the Deputy Director for the DC Promise Neighborhood Initiative (DCPNI). At DCPNI, Isaac led efforts to improve outcomes in the Kenilworth-Parkside community in Ward 7 of the District of Columbia through the strategic coordination of programmatic solutions and research-based strategies. Prior to his time at DCPNI, Isaac served as a Senior Research Scientist at Child Trends where he worked with nonprofits throughout the United States on the development and modification of performance management systems and evaluation designs. In addition, Isaac was also the Director of Learning and Evaluation for the Latin American Youth Center (LAYC) where he led the organization’s evaluation and performance management work.
You can read interviews with other social change leaders here.
Nell: You have spent your career using data to improve the performance of the nonprofits for which you worked. Why do you think performance management is so important for nonprofits? Do you think all nonprofits should pursue performance management? When does it make sense and when doesn’t it?
Isaac: I believe that every nonprofit should pursue some form of performance management because they owe it to the clients they serve. Most nonprofits will assume that they are making a positive difference in people’s lives, but in the vast majority of cases they are just guessing. Using some form of performance management will allow every nonprofit organization to confirm this thinking and to identify areas that can and should be improved so that the next cohort of participants can get better services than the last.
Unfortunately, one of the greatest challenges preventing a nonprofit from implementing some form of performance management isn’t a lack of resources, expertise, or time. It is fear. The fear that they will find out that their work isn’t having a positive effect. This fear is what nonprofit leaders need to overcome, not for the benefit of themselves or their organization, but because they owe it to the clients they serve today and the clients they will serve in the future. I believe that every nonprofit should strive to serve tomorrow’s clients better than today’s clients, and one of the only ways to ensure that this happens is the sustained use of performance management.
The type of performance management that each nonprofit should pursue should vary by the size and scope of their work. At a minimum, small nonprofits should be tracking basic demographic and attendance information on their participants, and hopefully at least one meaningful output or outcome. Whether this occurs in a computerized system or in a spiral paper notebook is up to the nonprofit. But it doesn’t have to be costly, and it doesn’t take expertise. It only takes the will and desire to improve as a nonprofit.
Nell: In the nonprofits in which you’ve worked how have you been able to secure resources to fund performance management? What is the case you and your colleagues have made to funders and what do you think it will take to get more funders investing in performance management?
Isaac: Raising funding for performance management work usually takes a mix of several different strategies and approaches for potential and existing funders.
First, I strongly encourage nonprofits to include some percentage (1 to 5 percent – possibly more) of funding in each grant submission or proposal dedicated to supporting performance management and outcome measurement work. By placing this small percentage into each proposal, a nonprofit can begin to raise funds for internal evaluation and performance management activities. It may not seem like a lot, but it can add up, and eventually generate enough funds for a half-time or full-time position to support in-house performance management work.
Second, I also strongly encourage nonprofits to engage in regular ‘funder education’ – where a nonprofit proactively meets with their funders to have ongoing conversations about outcome measurement and evaluation. This allows both the funder and the nonprofit to come to agreement on measurement expectations and to ensure that both groups are focused on the same concepts. I often suggest that the first of these types of meetings focuses on each group’s definitions of three commonly misunderstood terms: outputs, outcomes, and impact.
Finally, I would recommend that the nonprofit and funder have an honest discussion regarding expectations of results and the funding necessary to support the related evaluation work. If a funder is expecting an random control trial (RCT) to be completed to determine ‘impact,’ then the nonprofit should be willing to push the funder to support a large investment to pay for a high quality evaluation. If the funder is only willing to support a small amount for outcome measurement, then the nonprofit should clearly articulate what is possible.
Nell: Ken Berger and Caroline Fiennes recently argued that we may have gone too far by asking nonprofits to produce research about their own outcomes. What’s your response to that argument?
Isaac: I fully support Ken and Caroline in their argument that most nonprofits should stay away from trying to produce impact research. The desire for ‘impact’ is something that has been (and continues to be) pushed unfairly (and without financial support) by the funding community.
I honestly think a lot of confusion in this space comes from inconsistent use and understanding of the term ‘impact’. The term ‘impact’ has a precise definition among researchers but is often used in a much broader context among funders, nonprofits, and the general public. In the research and evaluation world, impact is used to describe the effectiveness of a program while eliminating as many potential confounding factors as possible. That is why the use of random control trials (RCTs) is usually the cornerstone of impact research – RCTs are the easiest way to control for and eliminate confounding factors.
When most non-researchers use the term ‘impact’ however, they are usually just asking if the program or organization works and if it is making a difference for its intended service population. That is a much lower bar to set, and yet it is a critical distinction in this discussion. If you are thinking about ‘impact’ as a researcher, you will need a large amount of resources and expertise to determine ‘impact,’ which usually means completing one or more formal evaluations. If you are thinking about ‘impact’ in the more general sense and less strict way, then pursuing some form of performance management system will allow a nonprofit to determine if their efforts have been successful.
I do think every nonprofit should pursue some form of performance management to ensure that their work is having a positive effect as a complement to existing research that others have done. Relying only on the use of others’ research does not guarantee that a nonprofit will provide effective services and achieve positive outcomes. This type of research is a like a recipe – it shows what has worked in the past and provides a guide for the nonprofit – but a recipe can still be ruined with poor implementation or planning.
Every nonprofit has an obligation to the people they serve (and not to their funders) to ensure that their programming is having a positive effect (or at the very least not causing harm). Without some form of performance management system in place (even one that just uses paper and pencil), a nonprofit will never know if they have strayed too far from the recipe provided by previous research.
I also think there are a growing number of very sophisticated nonprofits that should be using AND producing research on effective programs. Every year, I see more and more nonprofits that hire talented and unbiased researchers dedicated to internal evaluation and outcome measurement work. These individuals are just as talented and unbiased as their colleagues working in traditional research and evaluation organizations. They can, and should, produce original research that can help inform the nonprofit field. The real challenge comes in nonprofit organizations finding the resources to support the hiring and retention of these individuals. Not every nonprofit will have the resources or capacity to hire one or more of these individuals – but those that do should absolutely be trying to produce original outcome and impact research to provide ‘recipes’ for effective programming that nonprofits with fewer resources can use in the future.
Nell: Your former organization, DC Promise Neighborhoods, is part of the national Promise Neighborhoods Initiative launched by the US Department of Education in 2010 and modeled after the famous Harlem Children’s Zone. How successful has this national replication of a successful local model been? Have you been able to replicate outcomes? And what hurdles, if any, have you and other replication sites found?
Isaac: I think that there has been some initial success among the Promise Neighborhoods. Part of the challenge that all the Promise Neighborhoods face is that the Harlem Children’s Zone did not achieve their success overnight. They have been working in Harlem for decades, so it would be unrealistic to believe that the Promise Neighborhoods would be able to create large scale change in a matter of a few years.
However, there are signs of progress across all of the Promise Neighborhoods. Each of the Promise Neighborhoods started to address a few outcomes with the initial round of funding, and these outcomes varied. Some focused on math and reading proficiency for students, some focused on obtaining medical homes for young children, and others sought to increase the amount of healthy food consumed by residents. In DC, we focused on improving school attendance.
I do think that most of the 12 Promise Neighborhood Implementation grantees were able to make progress on the outcomes they identified as initial focus areas. However, the very nature of the work (creating community level change) doesn’t lend itself to the rapid accomplishment of multiple outcomes in a short period of time. Each of the Promise Neighborhoods had to prioritize certain outcomes for their respective communities, and only several years later are they able to claim success and begin to identify the next set of outcomes to be addressed. So while certain outcomes haven’t necessarily been replicated across all the Promise Neighborhoods, that is due to the differences in priorities and community conditions rather than any problem with the model itself.
Photo Credit: Venture Philanthropy Partners
May was about the “era of adaptation.” We are living in an age where change is a true constant, and we must adapt. We must adapt how we use technology, give money, get educated, use data, and the list goes on. It is an exciting (if sometimes overwhelming) time filled with opportunity.
Below are my 10 favorite social innovation reads in May. But, as always, add your favorites to the list in the comments below. And if you want to see my expanded list, follow me on Twitter, Facebook, LinkedIn, or my newest addition, Google+.
You can see the 10 Great Reads lists from past months here.
- The Era of Adaptation is upon us, so says Antony Bugg-Levine from the Nonprofit Finance Fund, and as such “adaptation requires nonprofits to invest in building and sustaining their organizations, not just running programs.” Amen to that!
- And how people give is definitely undergoing change. A really interesting article in the Wall Street Journal profiled Laura and John Arnold and their scientific approach to giving away their billions, while hoping to redefine philanthropy in the process.
- Google announced a new giving app that allows users to give $1 donations to nonprofits. Doesn’t sound like much, but nonprofits should keep an eye on this. As Google continues to be everywhere, this is an innovation where you may not want to be left behind.
- Warren Buffett and his sister Doris are doing something pretty interesting this summer. They are offering the first ever philanthropy MOOC (Massive Open Online Course). Over six weeks, participants will learn about philanthropy and then some participants will be given money to give away to nonprofits.
- As women increasingly control wealth, Anya Kamenetz from Fast Company asks the question, “Will Women Billionaires Make Better Philanthropists?”
- And then there’s technology and all that it is changing. Writing as a LinkedIn Influencer, David Kirkpatrick describes the coming of age of the Millennial generation and the opportunity (and burden) of deciding whether to use the gift of technology for the greater, or just their own, good.
- Big data has the potential to create enormous change as well. Regardless of your politics, Obama’s reelection team included some really great minds and one of them is now working on using big data to solve social problems.
- And how about higher education? Ben Thurman breaks down the growing innovations in higher education on the Dowser blog. From online courses, to apprenticing, to Silicon Valley’s growing interest in higher education innovations.
- Writing on the Stanford Social Innovation Review blog John Gillespie provides a very useful set of 5 questions nonprofits should ask themselves to determine if they are truly ready to scale.
- In a thought-provoking two-part series (here and here), Caroline Fiennes explains why nonprofits should monitor, but not evaluate, their work, and the role social scientists play in the evaluation of big ideas. Hers is a great distinction, but I’m not sure how we execute on the concept in the real world.
Photo Credit: AngryJulieMonday
In this month’s Social Velocity blog interview, I’m talking with Phil Buchanan. Phil is president of The Center for Effective Philanthropy (CEP) and was the first chief executive of the organization. Under his leadership, the organization has grown into the leading provider of comparative performance data to large foundations and other grantmaking institutions. Phil also serves on the board of Great Nonprofits and is a columnist for The Chronicle of Philanthropy.
You can read past interviews in our Social Innovation Interview Series here.
Nell: At the Center for Effective Philanthropy you work to make philanthropists more effective at creating social change, but a large part of philanthropy is driven by emotion and passion as opposed to results and data. How do you reconcile a push towards more reasoned philanthropy with the emotional aspect that will always be present?
Phil: I understand that some people feel this tension, but to me, it’s hard to understand because I think emotion and passion and results and data can – and should – cohabitate very happily. The passionate, emotional desire to make change is what inspires the commitment to get results. If you believe deeply in helping people in need, but do it in a way that doesn’t help, what kind of emotional satisfaction do you get from that?
Fay Twersky of the Hewlett Foundation articulated this very well in an essay in Alliance Magazine. She says impact should be pursued with “a warm heart and a hard head.” I like this way of thinking about it.
Nell: One of the the things you promote at CEP is a move from evaluating nonprofits based on overhead spending to evaluating them based on achievement of results. But sadly most funders haven’t yet embraced this distinction. What will it take for funders and the general public to recognize that overhead percentages are meaningless and destructive to the nonprofit sector?
Phil: I think the adoption of better nonprofit performance assessment practices is part of the answer. The more data nonprofits can point to that can show what they achieved with their total budgets, the less relevant how that budget was divided will feel to donors.
Look, I think people tend to gravitate toward that which is available, quantifiable, and comparative. Overhead percentages are all of those things, so they become the default performance measure even those they don’t tell you anything about performance. Caroline Fiennes of the U.K. has a great new book called It Ain’t What You Give, It’s the Way You Give It, and one of the best parts is that she really slays the argument for looking at administrative costs, while also providing guidance on how to approach performance measurement.
The rub is that the only way we’ll get better overall nonprofit performance assessment practices is if funders support that work. In our research, we have seen that, contrary to the stereotypes, nonprofits care about assessment and are working on it. But they want and need much more support – financial and non-financial – from their funders. I hope that funders embrace this and support better assessment practices in service of better outcomes.
I think Mario Morino has been a powerful voice on this topic and I recommend his book, Leap of Reason, to everyone I can. I hope people are listening to Mario because measuring effectiveness isn’t some academic issue. People who work at nonprofits deeply want to be effective. Foundations want to be effective. The people we help desperately need us to be effective. So we should – and we must – figure it out and get beyond empty measures. And many have. There are some fantastic exemplars when it comes to nonprofit performance assessment. But there are not enough.
Nell: In addition to leading CEP, you also serve on the board of GreatNonprofits, which allows individuals (clients, donors, volunteers) to review nonprofits. How does the idea of individual consumer reviews of nonprofits fit into the larger movement to evaluate nonprofits based on outcomes when the average person doesn’t yet understand or embrace the idea of nonprofit performance measurement?
Phil: In some ways I think it’s very easy for anyone to grasp. You’re trying to help someone; shouldn’t you ask whether they feel they have been helped? GreatNonprofits can provide that read on whether individuals served by a nonprofit feel they’ve been helped. I think GreatNonprofits, which Perla Ni founded and leads, is really important and I also think we need other kinds of efforts to collect and analyze beneficiary perception data. We need both the kind of open, web-based opportunity GreatNonprofits offers as well as rigorous, survey-based efforts such as the Center for Effective Philanthropy’s YouthTruth initiative, which helps schools, districts, and funders hear from middle school and high school students. We’re debating school reform in this country yet many of those with power and resources don’t understand the students’ experiences. We know that those experiences correlate to outcomes, so this kind of perceptual data could be a vitally important “leading indicator” of progress.
Nell: Philanthropy tends to be fairly risk averse and focused on program funding, as opposed to the organization-building capital investments (money to build organizations rather than buy services) the nonprofit sector so desperately needs. What do you think it will take to get more philanthropists to make riskier, longer-term, organization-building investments?
Phil: I think there needs to be a greater recognition that we count on organizations to get the work done. Sounds obvious, I know, but I think funders sometimes forget.
It is stunning, and sobering, that despite the valiant advocacy of Paul Brest, Paul Shoemaker, GEO, NCRP, and others, there has been no increase in the provision of general operating support over recent years. But we also need to be careful not to pretend operating support alone is the answer. Our research demonstrates that what really matters to grantees is operating support that is multi-year and a decent chunk of change – six figures or up in annual support, ideally. So the problem isn’t just one of grant type, it’s also one of grant size.
This comes back to assessment, too, in my view. If, as a funder, you know what you’re going after, and there is an organization that is focused on the same goal and can show that it’s delivering results, why would you not provide significant, long-term, unrestricted support? And, if you can’t find organizations delivering results toward your shared goal, why wouldn’t you fund in a way that would allow them to build that capacity?
Nell: You recently wrote a fairly scathing critique of Dan Pallotta’s new book, Charity Case because you thought his approach to advocating for the nonprofit sector was misguided. Yet the nonprofit sector is largely underfunded, undervalued, and dismissed in the broader regulatory and political environment. What do you think it will take to change that reality?
Phil: Pallotta’s book doesn’t advocate for the nonprofit sector that I know – or for one that I would ever hope to see. He wants the sector to become something entirely different, something a lot more like business, something that ultimately might not be discernible at all as a distinct sector. His take on the sector is both ahistorical (he demonstrates almost no understanding of the sector’s past contributions) and ideological (he has written that “the free market is a self-correcting system” that supports our “natural desire to help each other” and “only stops working when it is interfered with”). He is infatuated with free market analogies, believes financial incentives are the key to motivating people despite research demonstrating that they are not, insists that public trust in charities is lower than in other sectors when all credible research shows the opposite, and does not seem to understand that many nonprofits work to address the problems that exist as a result of market failures. His book is a disservice to the nonprofit sector.
So, then, what do we need to do to increase the appreciation of public and government officials for the nonprofit sector?
We need to start by standing up and asserting our value as a sector separate and distinct from business and government. We need to stop buying into the fiction that being effective means being “like a business,” whatever that even means. We need to stop praising the “blurring of the boundaries” and start articulating why we need organizations that pursue mission alone rather than profit for their shareholders. We need to explain why the sector is good for our society, good for business, good for government, good for citizens: we all need the nonprofit sector to be its best for us to be our best. And we need to re-learn our history – Olivier Zunz’s recent book on U.S. philanthropy would be a good place to start.
Yes, of course there is much work to do to improve the sector, but that doesn’t mean we need to tear it down. I wrote a series of blog posts for Duke University’s Center for Strategic Philanthropy and Civil Society a few years ago and argued that just as it is possible to walk and chew gum at the same time, it is possible to believe both that the nonprofit sector is and has been a defining strength of this country and that it must dramatically improve its effectiveness. It is possible to both celebrate the diversity of the sector and its various organizations and push for greater clarity of organizational goals, strategies, and performance indicators. It is possible both to applaud initiatives fostering “social innovation” and the government’s embrace of this push and also recognize what has worked in the past.
We need not tear down the sector to improve it. We need not disparage all that has come before in order to chart a better future.