Incorporating evaluation at the program design stage is an essential element of ensuring that evaluation activities will produce useful results. Planning an intervention and designing an evaluation strategy should be inseparable activities. To ensure the relevance and sustainability of evaluation activities, project designers, in collaboration with national and local stakeholders and collaborating donors, must work in a participatory manner to develop an integrated and comprehensive evaluation plan.

The purpose of this chapter is to provide practical guidance to HIV prevention program managers at various levels in developing realistic integrated and comprehensive monitoring and evaluation plans. Depending on whether such plans are designed for the national or project level, they will likely consist of widely different objectives, indicators, and methodologies to obtain those indicators. National and large geographic areas, such as provinces, may focus more on impact-related measurements, such as prevalence and behavioral estimates for HIV and sexually transmitted infections (STIs). Project-level evaluation may more appropriately concentrate on process and immediate outcome indicators, such as number of people reached and STI clients treated, as well as qualitative research methods to determine whether intervention strategies are appropriate for the target audiences.

Projects at all levels, whether they consist of multiple integrated projects or single interventions, should have an evaluation plan for assessing the progress of the program in achieving program goals and objectives and informing key stakeholders and program designers about the results of the evaluations. Such plans will guide the design of evaluations, highlight what information or data need to be collected, and how best to collect it.

Comprehensive evaluation plans should describe the overall purpose(s) of the evaluation, the specific evaluation questions to be addressed, the evaluation designs and methods to be used, what data are to be collected and how, the resources that will be necessary, who will implement the evaluation, and the basic evaluation plan timeline. They are often written to cover a 4- to 5-year period because they may involve multiple evaluative efforts on multiple interventions for multiple target populations, some of which require time to observe intervention or program outcomes (immediate or short-term effects), as well as overall program impact (long-term effects).

This chapter applies the conceptual evaluation framework offered in Chapter 1 and describes the steps involved in developing an integrated and comprehensive evaluation plan, the factors that may influence this process, and key questions that should be asked by every program.

Rationale For Developing An Evaluation Plan

It is often helpful in the beginning stages to review with all stakeholders the reasons for developing a comprehensive evaluation plan. Some of the benefits that can be derived from the evaluation planning process are:

• Evaluation planning will provide program managers and stakeholders alike with the opportunity to assess the evaluation needs, resources, capabilities, and priorities in their area.
• Having an evaluation plan will show stakeholders how the program plans to be accountable for the resources they have received.
• In the process of developing the evaluation plan, existing data sources and past or concurrent evaluation activities are often identified. Capitalizing on such existing data sources and past evaluative efforts can lead to a more efficient, less redundant plan for new evaluation monies.
• Having a long-term evaluation plan can clarify future decision-making regarding evaluation priorities.
• Finally, having a comprehensive evaluation plan in place may also favorably influence donor decision-making.

Key Elements Of An Evaluation Plan

In developing an evaluation plan, a planning group should focus on the following key elements:

• Scope of the evaluation—Specifying the goals and objectives of the program and developing a conceptual framework (or logic model) that integrates and correlates the inputs, activities, outputs, outcomes, and impact, and establishing realistic expectations for what the evaluation will provide or show
• Methodological approach—Developing an evaluation design, including specifying the outcome indicators or measures and the data source and plans for data analysis
• Implementation plan—Delineating activities, roles, and responsibilities, and a timetable for identified activities with realistic expectations of when data will be analyzed and results will be available
• Dissemination plan for the results—Determining who will translate the results into language that is useful to program designers, managers, and decision makers, how findings will be disseminated (for example, written papers, oral presentations, program materials), and implications for setting priorities for future evaluation activities

Developing An Integrated And Comprehensive Evaluation Plan

In some cases, these elements are already in place and the task of the evaluation planning group is to pull them together into an integrated logical whole. If these elements have not been previously developed, then the following steps may be helpful in beginning an evaluation plan.

Evaluation planning generally involves several common steps. These steps are described here in a general way so as to allow for differences among program types, target populations, stakeholders' needs for information, and the various levels such as a national program, a district or provincial response, and individual projects reaching specific target groups. Throughout the process, it is important to involve the various stakeholders: program planners, evaluators, Ministry of Health and National AIDS Control Program staff, as well as donors. Involving members of the target community also helps inform the evaluation planning process. This stakeholder involvement in the early phases helps to ensure that the evaluation results will be used in the end. As Box 2-1 shows, both internal and external perspectives are critical.

Step 1: Identify Program Goals and Objectives
The first step involves identifying the program goals and objectives and establishing a program logic model. This is done through writing a clear statement that identifies the program's goals and objectives (and sometimes sub-objectives) and describes how the program sees itself operating to achieve these objectives. When this is done, a program logic model can be easily diagrammed and used to establish an evaluation plan.

Each component of the program should have objectives and sub-objectives specified, and each objective and sub-objective should designate measures of success indicators that should be collected to determine progress. Table 2-1 illustrates a program logic model showing sample goals, objectives, activities, indicators, and methods to obtain indicators, at both the country and project levels.

This model illustrates the way that the role of national governments in monitoring and planning HIV prevention is distinct from, yet complements, the implementing strengths of individual projects at the local level. For example, individual projects do not typically conduct impact evaluation because their impacts are typically seen in tandem with those of other projects that work synergistically at multiple levels toward the same goals. Impact evaluation is most appropriately measured in large geographic areas, such as a province or country, and examines whether the collective efforts of numerous projects are reaching their desired impacts. These impacts can be measured through the serosurveillance systems, which monitor trends in HIV and STI prevalence as well as through repeated behavioral risk surveys. Local organizations, which have direct contact with target groups through outreach, STI treatment, and care, for example, should focus their evaluation efforts on assuring quality program implementation as opposed to outcome or impact evaluation. This means more of a concentration on quality inputs, such as training and pretesting of communication messages, but also includes the coverage and density of interventions.

This model also illustrates the length of time needed to show progress at various levels—ranging from several months for process-level accomplishments, such as staff trained, to several years for outcome- and impact-level goals.

If the information or data necessary to complete Step 1 are not readily available, then plans should be developed to collect them. This step concludes with a written specification of the program logic model, using the framework provided in Table 1-1 in Chapter 1 and the format of Table 2-1, and a general statement about priority evaluation questions to be addressed in the evaluation plan.

This last aspect—framing the evaluation questions and then setting priorities among them—is sometimes quite difficult when there are multiple stakeholders involved and resources, time, and evaluation expertise are limited. Thus, the evaluation questions may need revision later in the evaluation plan development process.

Step 2: Examine Existing Data and Past Evaluation Studies
The second step consists of identifying existing data sources as well as other evaluative activities that may have been done in the past, are ongoing, and/or may have been sponsored by other donors. At this step, evaluation planning teams should assess whether other groups are planning similar evaluations and invite them to collaborate in the planning and coordination.

Step 3: Identify Internal and External Evaluation Resources and Capacity
Identifying evaluation resources means not only identifying the funds for the evaluation, but also other types of resources, such as personnel experienced in evaluation to assist in planning and conducting the evaluation activities. It also means determining the program's capacity to manage and link various databases and computer systems.

Step 4: Determine Evaluation Questions, Their Feasibility, and Appropriate Designs and Indicators
In the fourth step, evaluation experts and program managers clarify the priority evaluation questions, appropriate evaluation designs, outcome measures or indicators, data needs, and the methods by which this information will be collected and analyzed. Practical ways for obtaining data and maintaining a data system that is sustainable and easily accessed should be discussed. An operational plan for the comprehensive evaluation plan should be developed at this step as well. Thus, this step should conclude with a revised written plan briefly outlining the evaluation questions and evaluation design, data collection methods and analysis plan, and overall timeline for the comprehensive plan.

Step 5: Plan for Disseminating and Using Evaluation Findings
The fifth step is not always performed, but should be because it is extremely useful in ensuring that evaluation findings are used to inform program improvement and decision-making. This step involves planning how evaluation results will be used, translated into program policy language, and disseminated to all relevant stakeholders and decision makers. It should also involve a feedback loop to the planners of the next evaluation and a feedback mechanism should be built in so that past lessons learned can effectively inform new efforts.

It is important that evaluation findings be translated into language that is useful for program designers in their efforts to improve programs and that they contain implications and actions needed, if appropriate, at national and/or local levels. This fifth step is often best conducted in collaboration with individuals or groups familiar with the intervention. Evaluation results, like all messages, must be marketed and "packaged" for their target audiences. (Chapter 13, "Effective Dissemination of Data Collection Results," provides useful guidance on ways in which programs can better carry out this step of the evaluation planning process.)

This step is often overlooked in the planning and implementation phase of the evaluation, and often only becomes an issue when there is a problem at the end of the evaluation and someone asks the question, "How has this evaluation plan been implemented and how have the results been used to improve HIV prevention programs and policy?" If there has been no plan for disseminating results and using the findings, this question often cannot be answered because the people involved in the evaluation have forgotten the details or have moved on. The lack of such a plan can undermine the usefulness of the evaluation and future activities. Inadequate dissemination also commonly leads to redundancy in evaluation efforts because others are not aware of the findings of previous evaluation efforts. It also reinforces a negative stereotype about evaluations, which is that they are not really intended to help improve programs. For all these reasons, programs should include a plan for disseminating evaluation findings in their overall evaluation plan.

Using Technical Support

After an evaluation plan has been developed following the above steps and including the essential elements described, program managers are faced with the question, "How are we really going to implement this plan?" Clearly, the success of the plan depends on the technical capacity of the program and its associated staff to carry out the evaluation activities. This invariably requires evaluation technical assistance. Evaluation expertise typically can be provided by individuals internal or external to the Ministry of Health or National AIDS Control Program. Such individuals usually have knowledge and training in the theories and methods of applied program evaluation. Potential evaluation experts might be found in planning and evaluation units in the Ministry of Health, local academic institutions, non-governmental organizations, and private consulting firms.

Technical capacity at the field level to design, implement, and maintain data collection systems is important to ensure the uninterrupted flow of consistent data. Participation of local people with a vested interest in the results should occur throughout the process of planning, data collection, analysis, and feedback.

Some traditional methodologies in evaluation, such as surveys on knowledge, attitudes, behaviors, and practices, may not always be appropriate for all evaluations. Technical guidance is especially important in complex evaluation designs, such as quantitative surveys where sampling designs, sample sizes, and questionnaire design must be carefully determined (see Box 2-2).

Implementing The Plan, Checking Progress, And Deciding When To Make Mid-Course Corrections

A comprehensive evaluation plan as described above has multiple components, some of which need to occur in a sequential fashion. Additionally, sufficient evaluation resources (both human and fiscal) for the entire plan are often not available at the launching of the plan. Thus, implementing components of the evaluation plan in phases is not only logical and practical, it is also essential.

Carrying out and completing components of the evaluation in phases provides the opportunity for the evaluation planning group to assess how well the evaluation plan is working. Sometime after the group completes the first or second evaluation activity, the plan may need some revisions or corrections. It is helpful for the group to spend time at this point ensuring that the plan is still useful. This process need not be very involved. The group can begin by discussing the following questions:

• Are the evaluation activities going as planned?
• Are the evaluation questions that were initially posed being sufficiently answered? Are other data needed to answer these questions? How can such data be obtained?
• Do the evaluation questions themselves need re-framing? Have other evaluation questions arisen since the initial planning that need to be incorporated into the plan at this point?
• Are there any methodological issues that need addressing or changes that need to be made to the evaluation designs? Are there any other factors, practical or political, that need to be considered in the evaluation activities yet to be implemented?
• Are any changes in the plan needed at this time? How will these changes be made? Who will implement them?
• Is the right mix of personnel and fiscal resources still available to carry out the rest of the evaluation plan?
• How are findings from the evaluation activities so far being used and disseminated? Does anything need to be done to enhance their application to programs?

This assessment is most helpful if it occurs annually throughout the plan. If the plan is kept current and relevant, it will help ensure the usefulness and quality of the remaining components. It will also foster the overall success and usefulness of all of the evaluation activities by the end of the project period. Reviewing the evaluation plan and doing mid-course corrections as needed also facilitates the connection between the evaluation activities and the programs as well as the design of subsequent plans.

Conclusion

Although projects may differ greatly in their scope, target audience, objectives, and activities, they all need to include realistic and comprehensive monitoring and evaluation plans. Incorporating evaluation in the program planning and design phase is essential to ensuring that evaluation activities will produce useful results. Involving national and local stakeholders and collaborating donors is an essential element of successful evaluation activities, as is broad dissemination of results.

This chapter has described a five-step process to guide program managers in developing evaluation plans that describe the purpose of the evaluation, the specific questions to be addressed, the evaluation designs and methods, the resources that will be needed, the people involved, and the timeline. It also has presented a program logic model that evaluation planners can use as a framework for laying out the evaluation plan. By following these steps, program planners can greatly enhance their chances of conducting evaluations that are useful, informative, and timely for HIV prevention efforts.

Recommended Reading

1. Academy for Educational Development. Critical issues in HIV prevention evaluation: a monograph. Washington (DC): AED; 1998. Funding provided by the Centers for Disease Control and Prevention.
2. Catania JA, Moskowitz JT, Ruiz M, Cleland J. A review of national AIDS-related behavioral surveys. AIDS 1996; 10 (Suppl A): S183-S190.
3. Centers for Disease Control and Prevention (CDC). HIV prevention evaluation guidance. Atlanta (GA): CDC; 1998.
4. Chelimsky E, Shadish WR, editors. Evaluation for the 21st century: a handbook. Thousand Oaks (CA): Sage Publications; 1997. p. 145-201.
5. Coyle S, Boruch R, Turner C, editors. Evaluating AIDS prevention programs. Washington (DC): National Academy Press; 1991. p. 102-123.
6. Cronbach LJ. Designing evaluations of educational and social programs. San Francisco: Jossey-Bass; 1982.
7. Fetterman DM, Kaftarian SJ, Wandersman A, editors. Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks (CA): Sage Publications; 1996. p. 100-207.
8. Patton MQ, editor. Utilization-focused evaluation. 3rd edition. Thousand Oaks (CA): Sage Publications; 1997.
9. Rossi PH, Freeman HE, editors. Evaluation: a systematic approach. 5th edition. Newbury Park (CA): Sage Publications, Inc.; 1993.
10. Rugg DL, Buehler J, Renaud M, et al. Evaluating HIV prevention: a framework for national, state, and local levels. Am J Eval 1999;20(1):35-56.