Measuring sexual behavior resulting from HIV/AIDS prevention programs is a difficult undertaking. With its emphasis on evaluation beginning with project design, the AIDSCAP Project offers a unique opportunity to test innovative approaches to evaluation.
In 1996, the AIDS Control and Prevention (AIDSCAP) Project began the final phase of an ambitious effort to evaluate the progress of its HIV/AIDS prevention programs. As part of the final evaluation of comprehensive programs in 18 countries, tens of thousands of people will be interviewed and hundreds of group discussions held.
But the sheer quantity of the data is only one of the challenges facing AIDSCAP evaluation specialists. For the past decade, HIV/AIDS program evaluators throughout the world have been wrestling with some difficult questions. How can they track changes in people's most private behavior? Without reliable HIV incidence data in most countries, how can they be sure that prevention interventions have had an impact? And, when changes in behavior or disease rates are detected, can they be attributed to the actions of one program or project?
AIDSCAP evaluation specialists started searching for answers to these questions when the project began in 1991. Specific objectives and ways of measuring whether they have been met are built into the design of every country program and each of the "subprojects" that comprise that program.
"Most evaluation processes are post hoc," said Dr. Susan Hassig, former associate director for evaluation at AIDSCAP. "Considering evaluation in the design of the project at all levels shouldn't be innovative, but unfortunately, it is."
Although monitoring and evaluation have been part of the project from the start, AIDSCAP's approach to evaluation has evolved over the past five years as more has been learned about the complex behavioral, social, economic and cultural factors that drive the HIV/AIDS epidemic -- and about the limitations facing evaluators.
Measuring Success
Most disease control programs use biological indicators to monitor and evaluate progress. But for HIV/AIDS, biological data are either unavailable in most of the developing world or of limited value for evaluating progress against the epidemic.
Some developing countries have fairly accurate information on the level of HIV infection in subsets of their populations at a given time. However, because these prevalence data can represent infections that occurred six months to ten years earlier, they do not necessarily reflect change or lack of change in risk behavior.
Rates of other sexually transmitted diseases (STDs) can be proxy indicators of changes in sexual behavior but must be interpreted with caution. STD statistics seldom capture all or even most STD cases because many people seek treatment outside the formal health sector and many health information systems are weak.
The most useful kind of biological data for evaluating the impact of HIV/AIDS prevention interventions are rates of new HIV infections. Such incidence data are rarely available because it is so difficult and expensive to collect them.
Demonstrating a meaningful reduction in HIV/AIDS transmission would require large sample sizes and high initial incidence rates in the target population, explained Dr. Daniel Tarantola, director of the International AIDS Program at the Harvard School of Public Health's François-Xavier Bagnoud Center for Health and Human Rights. Given all the constraints, he believes that using biological indicators to measure the impact of most HIV/AIDS programs is "both unrealistic and impractical."
Assessing Behavior
In the absence of reliable biological data on HIV/AIDS and STDs in most countries, behavioral indicators are the most important measures of progress in HIV/AIDS prevention. Because people throughout the world find it difficult to talk about their own sexual behavior, evaluating whether behavior change is occurring requires creativity and patience.
Like most health campaigns to change behavior, HIV/AIDS prevention programs must rely on people's own reports of their behavior. Although concerns about the accuracy of data on self-reported behavior is warranted -- particularly when the subject is sexual behavior -- it is possible to design internal and external checks to ensure that the information is valid and reliable. Examples include phrasing the same question several different ways, using different methods to collect the same information and, where appropriate, comparing respondents' answers with those of their partners.
A more difficult challenge for HIV/AIDS evaluators is detecting whether any behavior change has occurred over a relatively short period. A typical prevention project lasts two to five years, but the experience of anti-smoking and family planning campaigns has shown that behavior change can take decades.
After five years or less, "we may not see huge amounts of change," noted Dr. Jan Hogle, AIDSCAP evaluation officer. "Change occurs in incremental steps, and we need to figure out how to capture those intermediate steps."
Making a Difference?
Ultimately, the question everyone is most interested in answering -- "What difference did the intervention make?" -- may be the most difficult one to answer. Because behavior is influenced by so many factors, it is impossible to attribute changes in behavior to the efforts of one program.
Attribution is also difficult because the epidemic has its own dynamic. "You don't know what would have happened without your intervention," said Dr. Tobi Saidel, AIDSCAP evaluation officer.
The only way to overcome these difficulties -- a study with a randomly selected control population that does not receive any prevention services -- is an expensive undertaking that does not always reflect the reality of an actual prevention program, not to mention the ethical dilemma it poses.
"Every donor, every NGO, every implementor of an HIV prevention project wants to know that his or her project is having an effect -- that's human nature," said Stephen Mills, epidemiologist and evaluation officer in AIDSCAP's Asia Regional Office in Bangkok, Thailand. "Unfortunately, implementing the kind of rigorous study that has the power to say that your intervention has had that effect could double or triple the cost of the intervention."
To demonstrate an association between AIDSCAP interventions and behavioral or HIV/AIDS trends among the project's relatively small target populations would require a particularly complex study, noted Dr. Tarantola, who served on an independent team that conducted AIDSCAP's midterm evaluation for the U.S. Agency for International Development in 1994-95. "The danger is that the research objective would supersede the prevention objective," he added. "AIDSCAP's focus is naturally on prevention."
What AIDSCAP and other project evaluators can do instead, explained Dr. Thomas Rehle, AIDSCAP's new associate director for evaluation, is use multiple methods of evaluation. The information from these sources can be "triangulated" to gain as complete a picture as possible of what is happening in the field. AIDSCAP uses a variety of methods to achieve this goal, including process monitoring, surveys, focus group discussions and in-depth interviews.
Monitoring the Process
The most basic form of evaluation is process evaluation -- monitoring whether a project is doing what it is supposed to do. AIDSCAP developed a standard set of indicators, such as number of training sessions held, number of people educated and number of condoms sold or distributed, and a decentralized reporting system to monitor implementation of each subproject and country program.
The almost 300 agencies that implement AIDSCAP subprojects submit monthly "process indicator forms" (PIFs) to report on their progress toward reaching the targets specified in their project plans. These standardized forms make it possible to aggregate seemingly disparate data from many projects for program-wide reporting and feedback to the field.
At first field staff found this system too time-consuming, but many project managers now consider the PIFs a valuable monitoring tool. In fact, when evaluation associates from AIDSCAP country offices were given the option of reporting quarterly rather than monthly process data, they said they preferred to submit monthly reports.
Understanding the Numbers
AIDSCAP evaluation specialists monitor biological indicators when data are available, but rely primarily on behavioral indicators to track trends that reflect the influence of a variety of interventions. For each program and subproject, they look at changes in behavior known to reduce the risk of HIV transmission, such as consistent condom use and having fewer sexual partners.
Quantitative information on behavior is collected primarily through knowledge, attitudes, beliefs and practices (KABP) studies conducted among target audiences. To gain a better understanding of this information, AIDSCAP also uses methods that produce more qualitative information, such as in-depth interviews, focus group discussions and observation.
"Generally people think of evaluation as numbers and percentages," Dr. Hassig noted. "What we've found is that especially with HIV, the qualitative information is extremely important to understanding what the numbers actually mean."
AIDSCAP's emphasis on qualitative data is one of the most exciting aspects of the project's evaluation strategy for Maxine Wedderburn of HOPE Enterprises, the private research organization responsible for monitoring and evaluating the AIDSCAP program in Jamaica. "There is such a richness coming out of the qualitative work," she said, noting that many of her company's clients consider qualitative research a "luxury" they cannot afford.
Wedderburn believes that using qualitative methods is particularly important in HIV/AIDS evaluation because of the sensitive nature of the subject: personal sexual behavior. In-depth individual and peer group discussions allow time for participants to develop confidence and trust in interviewers or facilitators. The result, she said, "is more rapport -- which leads to more disclosure."
Qualitative information can help evaluators understand why certain interventions were effective and others were not. It can identify barriers to behavior change and suggest ways to improve prevention efforts.
In Haiti, for example, women factory workers talked about using family planning as a strategy for discussing condoms with their husbands. And conversations with Haitian men suggest they were willing to talk to their wives about condoms because the women had learned how to use them from an HIV/AIDS prevention project -- rather than from another man.
In Nepal, qualitative research led to an important change in the AIDSCAP program plan. The original plan was to reach two important groups -- truck drivers and commercial sex workers -- in towns along major highways. But interviews and a mapping exercise revealed that the truck drivers prefer to patronize sex workers in tea shops, restaurants and lodges in remote areas where it is easier to park and the police are less likely to interfere. As a result, project activities targeting truck drivers and their partners center around the smaller commercial centers and rest stops the truck drivers marked on road maps of Nepal.
Beyond PIs
The starting point for many AIDSCAP evaluations is the priority indicators (PIs) it helped the World Health Organization develop to encourage standardized reporting on HIV/AIDS prevention projects worldwide. In most cases, since these indicators are basic measures for national programs, they have to be adapted for AIDSCAP's smaller-scale interventions with more specific target audiences.
AIDSCAP is also developing indicators that are more gender-sensitive to gain a better understanding of the impact of HIV/AIDS prevention interventions on women. For example, two of the ten priority indicators are designed to track changes in condom use with "nonregular" partners and in numbers of nonregular partners. But there is increasing evidence that many women who have only one sexual partner are at risk of HIV infection as a result of that partner's behavior.
Instead, to assess HIV/AIDS prevention interventions with these women, AIDSCAP prefers to ask them whether they think their husbands have other sex partners. Women who answer yes to this question are then asked about their perceptions of personal risk, condom use and other changes in sexual behavior, including whether they are able to negotiate condom use with partners.
Another limitation of most measures of the impact of prevention interventions is that they focus on knowledge of HIV/AIDS and adoption of preventive behaviors. Experience has shown that knowledge is only the first step in a gradual process that leads to behavior change. Most evaluation efforts miss the steps in between.
To capture those small but critical changes in people's behavior, AIDSCAP evaluation specialists have developed a list of intermediate behavioral indicators along a continuum from acquiring knowledge to adopting and maintaining a new behavior. The indicators include perception of individual risk of HIV infection, intention to do something to reduce that risk, and having discussed HIV/AIDS and other STDs with sexual partners.
These intermediate behavioral indicators re-emphasize the importance of information that is already available, explained Joseph Amon, AIDSCAP associate evaluation officer. "We are collecting the information, but people aren't always looking at those questions," he said. "They're looking at the end-stage indicators."
During the recent evaluation of AIDSCAP's program in Haiti, this emphasis on intermediate indicators proved a useful way to detect progress toward behavior change among populations with high knowledge of HIV/AIDS and increasing but low rates of condom use. Evaluators found that indicators such as the ability to discuss HIV/AIDS with sexual partners and the ability to discuss condom use may be good predictors of behavior change.
In addition to shedding light on progress in encouraging people to change their behavior, careful examination of intermediate indicators provides valuable information for future HIV/AIDS prevention efforts. For example, the Haiti results suggest that those who have high knowledge of HIV/AIDS and confidence in their ability to negotiate condom use but are still not using condoms do not perceive themselves to be at risk.
"The intermediate indicators allow you to see where the obstacles are," Amon said. "And that's important because you need to be able to adapt your messages to those subsets of the population who haven't responded to your message yet."
One of the most promising ways to monitor and assess progress toward behavior change, according to Dr. Tarantola, is the behavioral surveillance survey AIDSCAP has conducted in Bangkok. These periodic surveys of sample groups of populations targeted by the HIV/AIDS prevention program in Thailand have yielded useful insights for improving interventions and tracking trends in high-risk sexual behavior.
The AVERT Model
Another tool to help program managers and evaluators understand the impact of a variety of influences without conducting studies to measure HIV incidence is the AVERT model developed by AIDSCAP. This mathematical model uses information on HIV and STD prevalence and on behaviors such as condom use and numbers of sexual partners among pairs of target populations -- young women and older men, for example, or sex workers and their clients -- to estimate the number of HIV infections that could be averted through interventions and the resulting changes in behavior or STD rates.
For example, the model shows that increasing consistent condom use from 10 to 42 percent among men and their sex worker partners in an area where HIV prevalence is 5 percent for men and 15 percent for sex workers would reduce the number of HIV infections that could be expected to occur in a year by 33 percent.
Such projections can help program managers decide where to target their interventions. They will also be helpful to HIV/AIDS advocates, Dr. Hassig points out, because they "show what it takes to stop the epidemic."
Given the complexity of HIV/AIDS evaluation, many have pointed to the need for one simple statistic -- a "super-indicator" similar to family planning programs' estimates of "couple-years of protection" -- to demonstrate to policy makers, donors, program managers and field staff that HIV/AIDS prevention works. The estimates produced by AVERT are a possible candidate for such an indicator. However, because of the diversity of HIV/AIDS prevention programs, a single indicator that could serve as a common measure for all programs may not be feasible.
Building Capacity
Assessing efforts to build the capacity of local organizations to sustain long-term prevention interventions is one of the most important -- and challenging -- tasks facing AIDSCAP evaluators. "Capacity building means many things to many people," said Kathi Kotellos, AIDSCAP associate evaluation officer. "It's difficult to define and therefore difficult to evaluate."
AIDSCAP looks for improvements in technical, managerial and administrative skills and systems and in the ability to communicate and collaborate with other organizations. The project uses qualitative and quantitative methods to monitor and evaluate capacity building, including process data collection, in-depth interviews with AIDSCAP country program managers, case studies and rapid organizational assessments.
With all of these methods, AIDSCAP collaborates with its implementing organizations to assess progress. "We want to find out -- from the organization's perspective -- what has changed," Kotellos explained.
Many of these organizations are interested in strengthening their evaluation capacity. By involving implementing organizations in every stage of the process, beginning with project design, AIDSCAP works with them to enhance their skills and promote a better understanding of the value of evaluation.
"Evaluation is often used as a sword over someone's or some project's head," Dr. Hassig said. "We've tried to emphasize a learning perspective and to create ownership, so people in the field aren't afraid of the evaluation process because they're part of it."
Designing and implementing AIDSCAP's evaluation plan has also been a learning experience for AIDSCAP's evaluation staff, according to Dr. Hassig. "I think we've demonstrated that evaluation is an extremely complex process that should be approached with respect and used where it is probably most instructive -- to make programs better."
-- Kathleen Henry
