This publication documents the experience of the world's largest international HIV/AIDS prevention project, which was implemented by FHI and its partners in 45 countries. It describes lessons learned during AIDSCAP, with examples and project profiles, in 10 technical and programmatic areas: behavior change communication, improving prevention and treatment of sexually transmitted diseases, prevention marketing, policy development, behavioral research, evaluation, gender and HIV/AIDS, management, AIDS care and support, and cross-border interventions.

Table of Contents

Making Prevention Work

1. Behavior Change Communication: From Individual to Societal Change

2. Improving STD Prevention and Treatment

3. Prevention Marketing: Condoms and Beyond

4. Policy Development and HIV/AIDS Prevention: Creating a Supportive Environment for Behavior Change

5. Behavioral Research: Using Results to Design Behavior Change Interventions

6. Evaluating HIV/AIDS Prevention Programs: Developing New Tools for Meaningful Measurement (See Below)

7. Women, Men and HIV/AIDS: Building Gender-Sensitive Programs

8. Managing HIV/AIDS Programs and Building Capacity to Sustain Prevention Efforts

9. Prevention and Care: Mutually Reinforcing Approaches

10. Crossing Borders: Reaching Mobile Populations at Risk

Partners and Acronyms

6. Evaluating HIV/AIDS Prevention Programs: Developing New Tools for Meaningful Measurement

One of the greatest challenges in HIV/AIDS prevention is determining what impact prevention efforts have had on the epidemic. Evaluators must track changes in people's most private behavior, assess program impact in environments where sexual behavior is influenced by a variety of factors, and develop evaluation measures that are reliable, valid and meaningful.

Early in the epidemic, it was assumed that biological indicators could be used to evaluate HIV/AIDS prevention programs. Many evaluation plans called for collecting data on the incidence of HIV and other sexually transmitted infections, as well as information on self-reported behavior, at the beginning and end of a program cycle. During the early 1990s, the World Health Organization's Global Programme on AIDS (WHO/GPA) developed a set of behavioral and biological prevention indicators for national AIDS control programs and standardized protocols to facilitate cross-country comparisons.¹

Experience with HIV/AIDS prevention has demonstrated that many of the early expectations about evaluation were unrealistic. Lack of resources has resulted in inconsistent collection of biological data, and pre- and post-project measures of behavior change have provided an incomplete and imperfect understanding of the impact of prevention efforts. As the epidemic and our understanding of the complex process of behavior change have evolved, evaluators have begun to develop more feasible and sensitive evaluation methods.

Refining Evaluation Methods

Evaluation was a key strategy for AIDSCAP from the beginning of the project in 1991. With its early emphasis on evaluation and the breadth of its experience, the project had an unprecedented opportunity to improve existing methods and test innovative approaches to evaluation.

It seems a truism that evaluation should be considered at the beginning of a project or program. In practice, however, this is not common. One of AIDSCAP's strengths was its incorporation of evaluation into the design of each country program. Each strategic and implementation plan for a country program, developed in collaboration with government and NGO partners and other stakeholders in the country, included a detailed evaluation plan that outlined the indicators to be used and how the data would be collected and disseminated. These customized evaluation plans designed for each of the 19 country programs and many of their "subprojects" were adjusted during the project as programs were revised and evaluation methods evolved.

Another unique feature of AIDSCAP evaluation strategy was its emphasis on diverse and complementary data collection methods. To an extent unusual for a large, donor-funded health program, AIDSCAP was able to complement quantitative process and behavioral data with more qualitative information from in-depth interviews, focus groups and rapid ethnographic studies. "Triangulation" of the results of quantitative and qualitative research yielded a wealth of information about the process of behavior change, the environmental factors that influence behavior, and how HIV/AIDS interventions affect knowledge, attitudes and behavior.

Tools and methodologies developed and disseminated by the project will ensure that other HIV/AIDS programs can continue to benefit from AIDSCAP's evaluation experience. These include a series of guidelines on different aspects of evaluation, such as incorporating evaluation into program design and conducting effective focus group discussions. Originally developed as references for AIDSCAP staff and partners in the field, these "Evaluation Tools Modules" have been used in HIV/AIDS programs, training workshops and university courses in many parts of the world.²

One of AIDSCAP's most important modules offers guidelines for conducting behavioral surveillance surveys (BSS), a methodology pioneered by AIDSCAP in Bangkok. Consisting of a series of repeated behavioral surveys in key target groups, the BSS enables national programs to track trends in HIV risk behaviors and to assess the combined impact of various HIV/AIDS interventions in a country. Inspired by the success of the surveys in Thailand, national and state HIV/AIDS control programs in Cambodia, India, Indonesia, Nepal and Senegal worked with AIDSCAP to establish BSS systems. A meeting of 28 experts from ten BSS projects in eight countries, which AIDSCAP convened in August 1997, produced recommendations for conducting these surveys worldwide (Box 6.1).³

In another emerging area of program evaluation -- capacity building assessment -- AIDSCAP developed and tested a methodology that includes instruments for organizational needs assessment and determining the outcomes of capacity building efforts. Capacity building evaluations in nine countries used a collaborative approach that emphasized self-assessment and use of the results as a tool for strategic planning.

AIDSCAP also collaborated with international organizations to advance the practice of evaluation in HIV/AIDS programs and provide guidance for prevention programs. For example, project staff worked with colleagues at WHO/GPA, USAID and the Centers for Disease Control and Prevention to develop the GPA prevention indicators. More recently, they helped the USAID Office of Population design a larger set of indicators for assessing the impact of interventions to improve reproductive health.⁴

Lessons Learned

• Involving project implementers in evaluation throughout a project encourages the use of evaluation data to improve programs and projects.

Deciding what information and how much data to gather in an evaluation involves difficult methodological decisions and trade-offs between the quality and utility of information. It is important to involve project staff in the evaluation process from the beginning to ensure that the research will produce data that are not only valid and reliable, but also useful for program planning.

AIDSCAP collaborated with indigenous NGOs and government ministries in the evaluation process, emphasizing the use of data to improve programs. In both Jamaica and Brazil, for example, biannual or annual evaluation meetings brought together representatives from the organizations implementing the AIDSCAP program in the country, the Ministry of Health or national AIDS control program, USAID, and the AIDSCAP office to review evaluation results. These meetings provided forums for discussing evaluation data and identifying ways to refine project or country program strategies based on the data. Frequent one-on-one meetings leading up to each review ensured that all participants were familiar with the evaluation results, and consultations between AIDSCAP's resident advisor and project staff after the meeting strengthened the recommendations and action plans adopted.

Prioritizing Research Designs

• Because resources for evaluation activities are limited, rigorous research designs are not feasible, or even appropriate, for every project.

AIDSCAP's recommended practices have evolved to reflect what is appropriate and possible at the national program level and the individual project -- or service-delivery -- level. This multilevel approach to prioritizing the degree of rigor needed for evaluation alleviates some of the tension that arises as a result of the sometimes conflicting evaluation needs of individual projects and national programs.

• At the service-delivery level, it is more efficient to limit evaluation activities to conducting formative research, monitoring process indicators and assessing capacity building efforts.

From the perspective of a national or regional program, it is not practical for every individual project to assess behavior change. AIDSCAP's experience with hundreds of projects showed that such assessments are time-consuming and require technical expertise that many service-delivery organizations do not have. Even when these organizations can collect and analyze data on behavior change, without an expensive study that uses control groups it is not possible to attribute changes that have occurred to the interventions of one project.

Only in the case of a demonstration project to test a new intervention or answer a research questions would there be justification for a more rigorous research design. Otherwise, when projects deliver services based on proven prevention strategies, the focus should be on ensuring that the services are delivered as intended. This can be done by tracking process indicators such as number of people trained, number of people educated about HIV/AIDS and number of condoms distributed. Projects that work to strengthen HIV/AIDS prevention skills should also assess whether they have succeeded in building capacity.

• In environments where many donors are supporting multiple interventions with overlapping target groups, certain types of evaluation are only appropriate at the national or regional level.

In such environments, it is impossible to attribute any changes detected by an evaluation to the efforts of a single project or organization. Therefore, it is more appropriate to combine the resources of national programs and donors to monitor national or regional trends in behavior among different target groups, condom availability and sales, STD case management, policy development efforts and epidemiologic impact.

In Senegal, for example, where AIDSCAP worked with 25 organizations, various target audiences were reached by many different interventions. So instead of trying to assess the contribution of each of its projects to behavior change in Senegal, AIDSCAP helped the Senegalese Ministry of Health develop a behavioral surveillance survey to track the combined effect of all HIV/AIDS prevention efforts on sexual behavior among target audiences.

Moving Beyond "PIs"

• Surveillance of trends in HIV risk behavior among specific population groups is an effective tool for monitoring and evaluating HIV/AIDS prevention efforts.

Most HIV/AIDS prevention programs measure progress toward meeting predetermined targets at the end of a project. For example, evaluators might look at whether a project has achieved a 30 percent increase in consistent condom use among youth in the project area. But setting such targets for expected behavioral outcomes requires precise estimates of baseline levels and an understanding of how much change is meaningful in each setting. And even when such targets are reached, the observed behavior change cannot be attributed to the activities of a single project.

The behavioral surveillance surveys methodology that AIDSCAP developed in Bangkok⁵ and later adapted in five countries offers a practical alternative for evaluating HIV/AIDS prevention efforts. It allows evaluators to monitor trends in HIV/AIDS knowledge, attitudes and preventive behavior over time rather than taking one end-of-project reading and measuring it against a somewhat arbitrary target. And, recognizing that attribution is rarely feasible, it looks instead at the combined effects of interventions on a national or regional level.

AIDSCAP found that the BSS, a series of cross-sectional surveys among different age, socioeconomic and occupational groups, is a particularly useful way of determining whether sexual behavior change is occurring in specific segments of the population. It provides more targeted information than systems that collect data only on the general population, and it ensures standardization, providing a degree of comparability that is rare when a number of different organizations are collecting evaluation data in a country or region. BSS also takes outcome evaluation to a more appropriate national or regional level, eliminating the need to collect data separately in a multitude of projects that reach the same target groups (Box 6.1).

• "End-stage" indicators measuring adoption of a preventive behavior, such as having fewer sex partners or consistently using condoms, do not adequately reflect the intermediate stages of sexual behavior change taking place among various target groups.

Early AIDSCAP evaluation plans called for the use of core indicators similar to those being developed by WHO/GPA to measure program impact on behavior in the general population. AIDSCAP used the basic constructs of these prevention indicators (PIs) but adapted them for specific target groups, such as youth, women, sex workers and men who have sex with men. The constructs focus on knowledge of prevention measures (an early stage of behavior change) and end-stage behavior changes such as partner fidelity and consistent condom use.

As it became evident that these indicators failed to address changes occurring in some groups, AIDSCAP added behavioral indicators that reflect intermediate stages of change along the continuum between knowledge and adoption of preventive measures. Quantitative and qualitative evaluation research in Haiti found evidence of important intermediate stages of behavior change, with less impact on end-stage behavior. For example, consistent condom use with a nonregular partner (as measured by WHO PI 5) did not increase substantially among workers who participated in a workplace education project. But the evaluation of the project did find significant increases in knowledge of HIV transmission and prevention methods and in the percentage of workers who felt confident discussing HIV/AIDS with their partners and negotiating condom use.

• Using a variety of methodologies and "triangulating" their results can help evaluators overcome many of the limitations they face in assessing the impact of HIV/AIDS interventions.

Valid assessment of the effectiveness of behavior change interventions presents numerous methodological and practical problems, including the bias inherent in self-reported data, the inability to attribute changes in behavior to specific interventions without a rigorous controlled study, and the insensitivity of HIV prevalence as an indicator of short-term behavior change.

A combination of quantitative and qualitative data is particularly helpful for assessing the complex and uneven process of sexual behavior change. Quantitative data on self-reported behavior may not provide convincing evidence of change in the short term (one to two years). Qualitative data gathered through interviews and group discussions can help evaluators detect movement in the direction of change that may not yet be discernible using the statistical techniques of knowledge, attitudes, beliefs and practices (KABP) surveys or seroprevalence studies. Qualitative data also enable evaluators to interpret the context in which behavior change occurs and helps program managers identify how to revise programs to reach and influence those who are not reducing their risk of HIV.

AIDSCAP used a variety of qualitative and quantitative methods to gain a more complete picture of the complex process of sexual behavior change. Qualitative data collected through focus group discussions, individual (key informant) interviews and rapid ethnographic studies were triangulated with quantitative data from KABP surveys or behavioral surveillance surveys. The addition of epidemiological data on HIV and other STDs, in the few cases where the appropriate data were available, enabled evaluators to compare trends in sexual behavior among target groups with trends in the epidemic among those groups (Box 6.2).

Assessing Capacity Building

• Progress in building the capacity of organizations can be measured using a combination of quantitative and qualitative methods.

A lack of consensus among HIV prevention organizations and donors on the appropriate indicators for evaluating capacity building has hampered the effective measurement of organizational change in the past. In addition, many organizations overlook the importance of baseline research in capacity building, and subsequently find it difficult to measure the extent, quality and types of the capacity that have been enhanced.

AIDSCAP developed multiple, complementary methods to monitor and evaluate capacity. These methods include organizational needs assessments, detailed inventories of the project's capacity building efforts, and a rapid organizational assessment that collected quantitative information on technical skill building, organizational management skill building, systems development, networking and sustainability. Organizations have used the results from these surveys to identify lessons learned and as the basis for strategic planning.

Measuring Potential Impact

• Models and other innovative evaluation methods can help evaluators gain a better understanding of program impact.

Given the difficulties and high costs associated with direct measurement of the impact of HIV prevention programs through large-scale incidence studies, evaluators are developing alternative methods of impact assessment. Their focus is establishing linkages between outcome data from program interventions and patterns of HIV prevalence and incidence. These methods fall under several categories, including application of simulation models, models to estimate HIV incidence rates and prevalence in selected populations, methodologies for linking behavioral and biological data, and tools for cost-effectiveness analysis.

AIDSCAP has created the AVERT model to estimate the impact of intervention outcomes on the number of HIV infections averted among the target population. These estimates provide a better understanding of the effect of current prevention strategies and can help program managers and other stakeholders set priorities for future HIV/AIDS programs (Box 6.3) .

Recommendations

• Evaluators should work with project staff and local stakeholders to match research methods to the nuances of particular evaluation questions and to the time and resources available for evaluation. Project and program managers should also establish mechanisms for assessing evaluation data at regular intervals and using those data to improve interventions.

• Evaluation designs should reflect what is feasible and appropriate for a project or program to measure. AIDSCAP recommends that small individual projects concentrate on evaluating service delivery and capacity building, leaving assessment of behavior change to national or regional evaluation efforts.

• In order to detect progress toward behavior change, HIV/AIDS programs should track intermediate indicators, such as the ability to negotiate condom use with a partner or perception of risk, as well as reported condom use and other "end-stage" indicators.

• HIV/AIDS programs should consider establishing behavioral surveillance systems to track trends in knowledge, attitudes and behavior among target audiences within the overall population. AIDSCAP's BSS methodology has proved an effective way of monitoring these trends and assessing the combined impact of various interventions.

• Because sexual behavior is an extraordinarily difficult area to assess, HIV/AIDS programs should use a variety of evaluation indicators and data collection methods. Triangulation of qualitative and quantitative data enables evaluators to interpret intervention outcomes and offers valuable insights into how to improve future interventions.

• Capacity building needs to be measured both quantitatively and qualitatively, and staff members from participating organizations should be directly involved in the process. Plans for evaluating capacity building should be built into the original design of a project or program to ensure that baseline data are available.

Future Challenges

Monitoring Sustained Change

HIV/AIDS prevention programs that have been operating for several years may find it increasingly difficult to detect changes in behavior because interventions have reinforcing rather than new effects. As a result, the potential size of changes in a target group will become smaller, and the sample sizes necessary to measure these effects will increase accordingly. Maintenance of reported behavioral change should receive greater emphasis in future evaluations.

Improving Data Quality

Data collection systems require substantial attention and maintenance to ensure the integrity of the data they provide. Active participation of key stakeholders is probably the single most important factor in ensuring that evaluation data will be reliable, valid, relevant and timely. Besides building local capacity to collect, analyze and disseminate evaluation data, an additional challenge for prevention programs is identifying and involving implementing partners who have a vested interest in the quality of evaluation results.

Evaluating Intervention Strategies

The Mwanza trial in Tanzania demonstrated that syndromic management of STDs in a population can reduce HIV incidence. A limited number of well-designed trials are needed to test the efficacy of other intervention strategies, particularly behavioral interventions to reduce sexual transmission of HIV and other STDs. These studies must be of sufficient size to yield clear results and should be designed to allow inferences about cause-effect relationships.

Linking Behavioral and Biological Data

Our understanding of how different behaviors and epidemiological factors influence epidemic patterns is still incomplete. There is an emerging consensus among evaluation experts that assessing the long-term impact of multiple HIV/AIDS prevention interventions requires investigation of trends in HIV infections along with trends in behaviors that may lead to infection. Political support and resources are needed to enable programs to collect and analyze HIV/STD surveillance data in combination with behavioral, socioeconomic and sociodemographic data.

References

1. Mertens T, Carael M, Sato P, et al. (1994). Prevention indicators for evaluating the progress of national AIDS programmes. AIDS 8:1359-1369.
2. AIDSCAP Evaluation Tools Modules Series. AIDSCAP/Family Health International, Arlington, Virginia.

   Introduction to AIDSCAP Evaluation (1993).

   Conducting Effective Focus Group Discussions (1994).

   Incorporating Evaluation Into Project Design (1994).

   Application of a Behavioral Surveillance Tool (1995).

   Qualitative Evaluation Research Methods (1996).

3. AIDSCAP/FHI (1997). Behavioral Surveillance Surveys (BSS): Issues and Recommendations for Monitoring HIV Risk Behaviors. Summary from the "Workshop on HIV Risk Behavioral Surveillance: Country Examples, Lessons Learned and Reecommendations for the Future," August 11-14, 1997, Bangkok, Thailand.
4. Dallabetta G and Hassig S, eds. (1996). Indicators for Reproductive Health Program Evaluation. Final Report of the Subcommittee on STD/HIV. The Evaluation Project/Carolina Population Center, Chapel Hill, North Carolina.
5. Mills S, Benjarattanaporn P, Bennett A, et al. (1997). HIV risk behavioral surveillance in Bangkok, Thailand: sexual behavior trends among eight population groups. AIDS 11 (suppl. 1):S43-51.