The following blog, written by Catalogue for Philanthropy President and Editor, Barbara Harman, was published in the Huffington Post on Tuesday, February 18th. It is the third post in a four-part series on the “evaluation problem.” Parts I and II can be found here and here.
“In Part II of this series, I noted that the watchdog organizations to which donors are typically directed do not, in fact, assess the quality of the programs that nonprofits create or the effectiveness of the work. Such organizations review finances, assess good governance, and provide valuable information, but they don’t answer the basic questions that donors should be asking. Is this nonprofit meeting a real need? Is it doing so with excellence? Is it having an impact on the community it serves?
“Impact,” of course, is the new “it” word — the gold standard (so we are told) for judging the value of social programs. Just check out the Office of Management and Budget’s webpage: according to the OMB the 2014 budget provides funding to increase “the use of evidence and evaluation to spread innovation and drive better results.” Nonprofits that don’t speak the language of results-oriented programming, or who don’t understand how to use data to represent results, may have a hard time holding on to the government support they currently enjoy. But it isn’t only the government that is asking the impact question. Watchdog groups like Charity Navigator are beginning to do so as well, and others will soon do the same.
Metrics measure something, but not everything. How do you measure the impact of arts classes on kids who live in homeless shelters? Or of sports programs that provide after-school options for young people who otherwise have none? If you can’t produce data that show these experiences lead to very concrete outcomes, does this make the experiences meaningless or imply that they have no result?
The truth is, it’s hard to measure most kinds of social value. Social problems are complex, and a good assessment has to begin with a good interpretation: what is the perceived need that an organization exists to address? What are the conditions within which it works? If I create a nonprofit designed to help young people graduate from high school and go on to college, it makes a difference where on the continuum these kids lie. Do they have parents who finished high school (or not)? Do they speak English as a first language (or not)? Are they attending high-performing schools (or not)? If I don’t look at the conditions, at the quality of the need, at the beginning of the story, then my measurement will be incomplete, misleading, and perhaps even wrong.
The point I have been trying to make in this and previous posts is that the work of evaluation is hard. But there is something about the language of impact assessment – using “evidence” to “drive results” – that belies the reality of this, that leads one to think the truth is entirely measurable, not at all in the province of human persons who are thinking and judging and analyzing. It’s no wonder so many nonprofits are hesitant about the process, feel excluded by it in advance, and worry that they won’t be able to meet its standards.
So we need broader and more complex kinds of measurements. But this is only part of the story. Nonprofits need to do important kinds of new thinking as well.
First of all, they need to articulate, examine, and codify their own beliefs, their ways of seeing the world. For a youth-serving arts program, this means articulating what the organization values and why it thinks these values have social meaning. The Boston Youth Arts Evaluation Project helpfully calls this identifying your “sacred bundle”: what do you think matters in the world? What are you trying to cultivate? What values underlie your work?
If you believe, for example, that creating art is empowering, that it generates in young people a grounded sense of self, a bond with adults, and, in turn, a positive connection with one’s community, then you know what you are trying to measure. Your “results” may have a different profile from those of an organization that seeks to reduce homelessness, or prepare adults for jobs, but what you are doing is still very significant: empowering disconnected youth is serious business. It just has a different way of talking about itself.
In other words, NEED and VALUE have to be part of the equation, both for nonprofits and for agencies and groups evaluating them. If we speak about impact without using these key words, without taking the measure of the world we inhabit, and without making judgments about what we value, our society and culture will be the poorer for it. We cannot allow this to happen.
Stay tuned for Part IV.”