October 2008

<< back to issue

Evaluation's Role in Creating Better Programs, part II

by Sharon Charnell Gherman, President/CEO of The Funding Exchange, Inc.

Last month, we learned how to determine whether an outside evaluator is needed for your project and what questions to ask yourself when designing an internal evaluation. Today we want to look further at the ways evaluations are used, how to create an evaluation, and what to do with the data you collect.

Types of Evaluations

Formal evaluation of programs and projects became popular in the 1960’s when a great deal of government money was funneled into grant programs in an attempt to solve social problems. Evaluation was seen as a way of documenting whether or not a program had been successful, and evaluation results were used to assist funders in choosing projects most likely to be successful in the future. The results were less than ideal, partly because grantors were attempting to use the scientific method to evaluate complex social issues, and the two really weren’t compatible.

Today there are several ways that project evaluations are used:

• Evaluations used to determine how successful a project has been. In this type of evaluation, data is gathered periodically throughout the project and a summary is completed at the end of the project period. Summary evaluations have been used most often by funders, and they are declining in popularity as funders focus more on creating lasting change (‘ capacity’) as a result of their grant programs.
• Evaluations used as a “snapshot in time”. This type of evaluation can assist project managers to identify parts of a project that are working or not working. They may use the assistance of an outside evaluator to make recommendations to help them make changes that improve their project as it continues. Snapshot evaluations compare ‘where we are now’ with ‘where we were before the project began’ as well as with ‘where we want to end up’ for comparison.
• Evaluations used to continually improve a program as the program progresses. This type of evaluation focuses on continually improving the projects results, which means that the project is tweaked as it goes along in response to results and feedback from participants and staff. It involves a high degree of internal cooperation, continuous feedback from participants and staff, and a programmatic ability to make changes to policy and procedures as the need arises.

Creating Evaluation Tools

Once you’ve decided the purpose of your evaluation, the key to designing effective evaluation tools lies in asking the right questions about the project. What information will you want to know? The type of information needed will determine the type of evaluation instrument you’ll use:

Numerical Tallies
You’ll hear this type of data referred to as “quantitative data”, which means that numerical tallies are used to document numbers within the project – how many encounters occurred during the project; how many participants of each racial group were served; how much of the project funds have we spent; or how much grades improved, for instance. These results are generally obtained by examining project records.

Surveys, Feedback and Evaluation Forms
This type of data is referred to as “qualitative data”, which means that the data is descriptive rather than numerical. Qualitative data gives context to the numbers, and are most useful for gathering subjective information from participants and staff. They are often ‘how well did we do’?, ‘what can we do better?’, and ‘how important is this?’-type questions.

If you have a number of participants from whom to gather information, consider some of the excellent online survey programs like Survey Monkey or Zoomerang. These programs summarize responses for you and make it easy to create professional-looking surveys, e-mail surveys to participants, and create charts and presentations of your findings. Most have the ability to use a free trial before purchasing a subscription, and the free trial may be sufficient to complete your project evaluation.

Surveys may utilize several types of questions:

• written responses (in order to summarize written responses, it’s necessary to categorize responses by their major theme)
• choosing one or several choices from a list of possible responses
• yes/no questions
• choosing a number to express the degree to which you agree or disagree with a statement (be aware that when designing a “how much do you agree/disagree” question, a scale of 1-6 will produce better answers than a scale of 1-5. The middle of a 1-5 scale is 3, which doesn’t indicate either positive or negative and thus gives you no information. The middle of a 1-6 scale is either 3, which is slightly negative, or 4, which is slightly positive.)

Interviews (phone or face-to-face)
Interview responses are very similar to surveys, but the interviewer records the answers instead of the respondent. Interviews work well when you have a small number of participants to work with, or as a follow-up with participants who haven’t responded promptly to surveys.

Next month in part III, we’ll discuss how to use data to improve your project.

The Funding Exchange, Inc. offers evaluation services and can assist you with your evaluation needs. We invite you to contact us at Sharon@funding-exchange.org if you have questions about project evaluation or if you may need the services of an outside evaluator.