Forward

Evaluation is too often an afterthought in the process of program implementation. This Handbook is dedicated to the premise that evaluation must be a critical part of the initial phases of planning effective HIV/AIDS prevention and care programs.

Readers of this Handbook will find that the authors have set the stage and provided the tools for a comprehensive and strategic approach to evaluation. The approaches they recommend yield useful and important information on the effectiveness of HIV/AIDS prevention, care, and support programs. Increasingly, the technical "know how" to develop effective interventions is available but the ability to demonstrate effectiveness is lacking. Moreover, the ability to increase and maintain global political will and resources directed to HIV/AIDS programs ultimately rests on our ability to show that our interventions can make a difference in reducing new infections and improving life for people infected with and affected by HIV.

Now more than ever, it is critical that we monitor and evaluate programmatic efforts to justify the expenditure of resources and enable them to grow to the levels that will ultimately have a sustained impact. Toward this end, Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries provides a comprehensive framework to help country program managers determine the effectiveness of their programs. This Handbook describes the methods needed to answer three simple but important questions: Are we doing the right things? Are we doing them right? Are we doing them on a large enough scale to make a difference? Answering these questions will allow program managers to decide how and when to modify existing programs or design new ones.

As we move into the third decade of this epidemic, it is incumbent upon us to demonstrate the results of our efforts. The third decade of the HIV/AIDS pandemic can be the decade of evidence of program effectiveness. These are times of great challenge and great opportunity. The time is now to show that we can make–and already have made–a difference.

Helene Gayle, MD, MPH
Director (1995-2001)
National Center for HIV, STD, and TB Prevention
U.S. Centers for Disease Control and Prevention

As the HIV epidemic continues its primary spread throughout developing countries of the world, the quality and effectiveness of interventions designed to reduce transmission are as critical as ever. After more than 15 years of fighting the epidemic, the worldwide track record is mixed: Some countries have documented success in curtailing new infections, while others continue to witness high incidence rates.

This Handbook is intended to fill an important unmet need in the field. Evaluating HIV prevention and care programs so that they can promote a high level of quality and effectiveness–even in resource-limited settings–is its subject. Just as substantial progress has been made in developing effective interventions in recent years, so have appropriate methodologies, strategies, and indicators for the evaluation of these programs been honed and sharpened. Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries brings together the combined experience of collaborations between scientists and program designers and implementers, universities and local community-based organizations, governments and private agencies–partnerships often hard won, but ones that have produced results.

Nevertheless, compared to other public health areas, the evaluation of HIV/AIDS prevention and care programs is relatively young, and critical questions remain on how best to measure their effectiveness. Vigorous, but healthy and necessary, debates will likely continue over the next years on evaluation and measurement issues. In describing various facets of evaluation issues, the chapters that follow here point out some of these continuing challenges.

The focus of this Handbook is on evaluating programs related to the sexual transmission of HIV. It was developed for a target audience of program managers and decision makers of service delivery programs as opposed to researchers who are evaluating the efficacy of interventions through experimental or quasi-experimental research methods.

Section I lays the foundation for HIV/AIDS program evaluation by describing the current consensus on generic concepts, approaches, and frameworks (Chapter 1) and by outlining the practical development of an integrated evaluation and monitoring plan for projects (Chapter 2).

Section II describes the operational approaches for evaluating the core program strategies necessary to effectively reduce the sexual transmission of HIV (Chapters 3, 4, 5, 6), illustrating the unique and often very disparate methodologies needed to evaluate different program components. The section concludes with long overdue guiding principles on how to evaluate HIV/AIDS care programs (Chapter 7). This may help to increase the commitment to funding care-related activities by providing donors and decision makers with the necessary feedback to determine whether the invested resources have yielded the expected results.

Section III focuses on measuring behavior change as the key outcome of standard prevention efforts. State-of-the-art methodologies and survey instruments for collecting behavioral data (Chapters 8, 9, 10) and assessing their validity and reliability (Chapter 11) are covered here, complemented by a chapter on tools for collecting qualitative information (Chapter 12). The section concludes with effective strategies for disseminating survey data to key audiences (Chapter 13).

Finally, Section IV tackles evaluation issues related to assessing program impact, emphasizing the need to analyze behavioral and sero-epidemiological data in tandem (Chapter 14). It also describes a newly developed tool for estimating the impact of different prevention strategies on HIV transmission (Chapter 15), and provides practical guidelines for cost and cost-effectiveness analysis (Chapter 16, 17).

The individual chapters in this Handbook attempt to follow the logic of a program evaluation cycle, and though they are written as "stand alone pieces," they are clearly interconnected. Cross-references to other chapters are found throughout the book to assist readers in understanding how the individual components of evaluation fit into the larger whole.

Introduction
Section I Role of Evaluation in HIV/AIDS Programs
Chapter 1: Conceptual Approach and Framework for Monitoring and Evaluation
Chapter 2: Developing an Integrated and Comprehensive Monitoring and Evaluation Plan
Section II Operational Approaches for Evaluating Intervention Strategies
Chapter 3: Evaluating Behavior Change Communication Interventions
Chapter 4: Evaluating Sexually Transmitted Infection Control Programs
Chapter 5: Evaluating Condom Programming
Chapter 6: Evaluating Voluntary HIV Counseling and Testing Programs
Chapter 7: Evaluating Care Programs for People Living with HIV/AIDS
Section III Methodologies for Measuring Behavioral Trends
Chapter 8: Uses of Behavioral Data for Program Evaluation
Chapter 9: Sampling Strategies for Monitoring HIV Risk Behaviors
Chapter 10: Indicators and Questionnaires for Behavioral Surveys
Chapter 11: Assessing the Validity and Reliability of Self-Reported Behavioral Data
Chapter 12: The Role of Qualitative Data in Evaluating HIV Programs
Chapter 13: Effective Dissemination of Data Collection Results
Section IV Assessing Program Impact
Chapter 14: Understanding HIV Epidemics and the Response to Them: Linking Data on Behavior Change and HIV Decline
Chapter 15: Translating Survey Data into Program Impact: The AVERT Model
Chapter 16: Guidelines for Assessing the Economic and Financial Costs of HIV/AIDS Prevention and Care Programs
Chapter 17: Guidelines for Performing Cost-Effectiveness Analysis of HIV/AIDS Prevention and Care Programs

The authors listed in this Handbook are only a handful of the individuals who have directly contributed to the conceptual and methodological advancements presented here from the field of evaluating and monitoring HIV prevention and care programs.

Since HIV was first discovered and interventions established to contain its spread 20 years ago, program managers and researchers alike have grappled with how to best evaluate and monitor interventions to contain the epidemic.

Our collaborators from numerous countries in the IMPACT Project, a U.S. Agency for International Development (USAID)-funded global prevention and care project managed by Family Health International, as well as the United Kingdom's Department for International Development (DFID)-supported HIV prevention projects in several countries, were generous in sharing examples of their monitoring and evaluation plans. We are also indebted to UNAIDS and the World Health Organization (WHO), which have consolidated the experience of partner agencies worldwide and promoted international best practice standards in monitoring and evaluation. Our evaluation handbook will complement their guidelines on indicators for national HIV/AIDS programs.

Additionally, we are grateful to our colleagues in other USAID-funded projects working in numerous countries–MEASURE, HORIZONS, AIDSMARK, and SYNERGY–who have tackled evaluation and monitoring issues from various angles and have contributed to many of the ideas expressed here.

Finally, all HIV program monitoring and evaluation efforts depend on the participation of people living with HIV and at risk of HIV infection. Whether through quantitative surveys, in-depth interviews, or focus groups, we ask our evaluation participants to share with us their personal behaviors regarding sex, injecting drug use, and related vulnerability–sometimes not just once, but several times. And they do. If not for the openness and candidness by the participants in these studies, our programs would not improve. We gratefully thank these participants who have represented the universe of individuals at risk of infection and hope that the operational approaches and data collection methods presented here lead to better services for the ultimate beneficiaries of HIV prevention and care programs.

The Editors

Contributors

This handbook has been funded by the United States Agency for International Development (USAID) through Family Health International's Implementing AIDS Prevention and Care (IMPACT) Project, Cooperative Agreement HRN-A-00-97-00017-00.

The authors and editors of Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries hope this book provides program managers and decision makers with clear and practical guidance on a subject that is critical to the continued success of efforts to reduce HIV transmission and contain the epidemic in the developing world.

ISBN 1-931547-01-07

The views expressed in this book by named authors are solely the responsibility of those authors and do not necessarily reflect the views of FHI or the United States Agency for International Development (USAID). The mention of specific companies or products implies no endorsement and does not suggest that they are recommended by FHI or USAID over others of a similar nature that are not mentioned.