Counting What Counts

How to Find Evaluation Consultants

If you are someone who believed in the yellow pages and uses the internet to identify a reliable plumber, the American Evaluation Association is the logical place for you to start looking for an evaluation consultant.  This is the professional membership organization for the tribe, and many leaders in the field cluster here.  If an evaluator lists themselves on this web site, at least you know they are somewhat "in the loop."  AEA even has a Find an Evaluator webpage organized by state.  What I like about this directory is that it most likely weeds out consultants who claim to also be qualified to provide program evaluation consultation but know very little about it.  In your ideal world, you will hire a consultant with training in program evaluation, and AEA is a great place to get such training.

Locally, there are all sorts of comparable online clearinghouses for consultants who serve the nonprofit community, and evaluation consultants often list themselves in these.  For example, I am listed in the consultants directory for the Northwest Development Officers Association, because I know that many Executive Directors in the Puget Sound region turn to this group with all sorts of questions.  Consider any directory that lists nonprofit consultants.   

Few of my clients find me through either of these resources.  Most ask trusted colleagues to make a recommendation.  This is probably the most tried and true way to find a good evaluator.

When to Hire an Evaluation Consultant

Does your organization collect hundreds of "end of program surveys" each year, only to leave then in a pile in a box waiting for someone to find the time look at them? 

Do you collect surveys year in and year out only to report essentially the same positive results to your funder, such as "92% of training participants say they liked the training."

If so, read on.

I am a firm believer in self-evaluation, or program staff designing and implementing their own program evaluation plan.  Good program managers want data on how their programs are doing and have the smarts to collect that information.  And, there are times when some strategic input from a consultant can be a wise investment.

The good reason to hire a consultant is to avoid investing time and resources into an evaluation that doesn't teach anything.  Wasting energy on collecting voluminous numbers of surveys when you don't have a system in place for analyzing the data is silly.  Continuing to collect information that doesn't teach anything new isn't something I'd recommend either. 

Consider hiring a consultant to review your internal evaluation plan and recommend cost-effective improvements.  For example, a consultant can help you devise a sampling system so that you can learn what you want to learn with fewer surveys.  A consultant can help you revise survey questions to uncover new things about the strengths of your program, and the opportunities for improvement.

Avoid costly consultants who don't leave behind useful tools and information.  Find a consultant who will increase your organizational capacity to use evaluation again and again, and say good-bye to the pile of surveys in box cluttering your office.

Why Evaluation, Why Now?

These days, great nonprofit organizations are interested in using program evaluation. Finding funding to support evaluation efforts, however, can be a challenge. The good news is that there are ways an organization can use evaluation tools without any outside assistance or funding. A low cost strategy for increasing evaluation in your own agency is to facilitate some evaluation planning internally.

Establish an evaluation planning committee that includes a few key people with varied perspectives on the organization and its programs (such as the ED, a program person, and a board member). In this small group, have a 30 minute conversation to determine why program evaluation is important to your organization. Here are some questions to help you get thinking, adapted from a useful memo called 9 Points to Consider Before Contemplating Evaluation.

• Why is it important to initiate an evaluation now?
• Who is going to use the evaluation information?  (e.g., executive director, board members, staff, members, clients?
• For whom are you completing the evaluation?
• -Do you expect an evaluation to help improve impact on program participants?
• -Do you expect an evaluation to change staff or program implementer practices?
• -Do you expect the evaluation to help change management policies or practices?
• -Do you expect the evaluation to assist fundraising?
• Looking at your answers to the 4 bullets above, which is most important for your organization right now? 

Once the group has reached consensus on the answers to these questions, you have the fodder for the first section of your evaluation plan. Be sure to take notes on your final decisions! In future entries, I will explain how to use the answers to these questions to hone evaluation questions and establish priorities, as well as to seek funding for evaluation.

Program Evaluation and Fundraising

On Monday, I sat in a large conference room filled with over 500 nonprofit fundraisers.  Who knew there were so many fundraisers in one city?  (To learn more about the field and such conferences, visit the Northwest Development Officers Association.)

Many nonprofits want to do program evaluation because they think it will help with fundraising.  But there is a big gap between doing an evalution and increasing donations to deliver meals to the elderly or computers to the community center.  Here's one tip for bridging that gap.

Monday's keynote speaker, Penelope Burk, shared research on what makes one time donors become repeat donors who increase their gift to your worthy cause.  Here's a major finding: after a donor gives a gift to your worthy cause, they want you to learn the measurable outcomes that the gift and your program produce. 

It is critical that you tell your donors how their gift is making a difference.

If you want to raise more funds, it's not enough to conduct program evaluation.  You should pull one of the tangible program outcomes revealed by the evaluation, and share it with your existing donors.

~Mary

Art with Heart

"I was always considered the artist in the family, until my sister had cancer."  I heard this statement as I sat in on an Art Buddy training this weekend.  This training, offered by Art with Heart, empowers local artists, writers and designers to connect with youth in need, to heal youth through creativity.  They have contracted with MEMconsultants to help plan program evaluation that assesses the impact of their programs.

"My sister started painting while undergoing chemo, and it simply changed her.  She's so much more loose now, she's always painting."  The speaker's smile communicated that her sister finds more life satisfaction after the cancer than before, thanks to painting.  It is this  quality of experience with art-making that Art with Heart fosters.  It is this kind of life changing shift that the ideal program evaluation would document and measure. 

Einstein is credited with saying "Not everything that counts can be counted, and not everything that can be counted counts."  The science/profession/industry of program evaluation has not yet figured out how to count the personal transformation Art with Heart aspires to catalyze and remain consistent to principles of scientific method and objectivity.  Evaluators such as myself try, and I believe it is worth Art with Heart's time and money to rigorously reflect on their program goals and find indicators that they are accomplishing these goals.  At the end of the day, in our training to measure what is measurable, will may overlook the most important "data" there is - the value of the smiles of someone whose sister is thriving.

~Mary

Welcome!

You've arrived at the very first blog entry for MEMconsultants. This blog will develop into a resource for youth programs interested in strengthening their programs and documenting their program successes.  It will also serve as a resource for those interested in donating to great mission-based organizations, but not sure how to pick the best ones.  Thanks for visiting, and come back soon.