|
Programs |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This report covers the FHI AIDS Control and Prevention (AIDSCAP) Project (1991-1997). Volume 1 covers regional program overviews, technical strategies, and program support strategies.
Volume 1 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Year/Category | KABP Surveys1 | FGD Studies2 | Studies Using Indepth Interviews | Assessed Capacity Building | Assessed STI Services (PI6 & PI7) | Condom Audits |
| 1992: | 8 | 3 | 2 | 0 | 0 | 0 |
| 1993: | 20 | 12 | 3 | 0 | 0 | 3 |
| 1994: | 22 | 15 | 6 | 0 | 4 | 7 |
| 1995: | 39 | 33 | 11 | 0 | 0 | 4 |
| 1996: | 61 | 58 | 26 | 5 | 8 | 6 |
| 1997: | 17 | 23 | 13 | 4 | 4 | 3 |
| Cumulative: | 167 | 144 | 61 | 9 | 16 | 23 |
1 Knowledge, attitudes, beliefs and practices
2 Focus group discussion
Lessons Learned and Recommendations
Prioritizing Research Designs
Evaluations of the effectiveness of HIV/AIDS prevention programs should examine short- and intermediate-term program effects (program outcome) and long-term program effects (program impact). Examples of program outcome and impact indicators for the different stages are illustrated in Table 21. For interventions designed to reduce sexual transmission of HIV, accurate knowledge about HIV risks, reduction of risk behaviors, and adoption of protective behaviors are considered appropriate short- and intermediate-term outcome indicators. Long-term effects include impact on HIV/AIDS trends, sustainability issues, and improved societal response.
| Program Outcome (Short- and Intermediate-Term Effects) | Program Impact (Long-Term Effects) |
| Changes in HIV/STI-related knowledge | Sustained changes in HIV/STI-related risk behaviors |
| Changes in HIV/STI-related attitudes | Changes in HIV/AIDS trends |
| Changes in HIV/STI-related risk behaviors | Improved capacity of community |
| Changes in STI trends (e.g., gonorrhea) | Reduced individual and societal vulnerability to HIV/AIDS |
| Increase in social support/community response | Sustained changes in societal norms |
| Changes in societal norms | . |
It is difficult to disentangle the net effects of a prevention program from the gross outcome and impact observed. Such estimates cannot be made with certainty, but only with varying degrees of plausibility. A general principle applies here -- the more rigorous the research design, the more convincing the resulting estimate. However, resources for evaluation activities are limited, and rigorous designs are not always feasible or appropriate for a particular project. From its experiences, AIDSCAP/FHI learned that different research designs are appropriate for different levels of evaluation, and the best practices for program evaluation have changed to reflect what is appropriate and feasible at these levels.
AIDSCAP found that a three-level framework for differentiating evaluation approaches was the most useful and efficient. These three levels are the subproject, country program, and global levels. The subproject, or service delivery level, does not require rigorous research designs to determine effectiveness unless a subproject is a new intervention or a response to an unanswered research question. At this level, evaluation research should be limited to process monitoring, capacity building assessment, and formative evaluation (when needed for project planning). Only in the case of a demonstration project, would there be justification for a more rigorous research design.
At the country program level, AIDSCAP carried out evaluation research on technical intervention strategies and policy and socioeconomic impact. One of the lessons learned at this level is that in a situation where multiple donors are conducting multiple interventions with overlapping target groups, certain types of evaluation are not appropriate at the subproject level, but are appropriate at the country program level. Areas for country-program-level evaluation include the technical strategies, including behavioral trend analysis of different target groups using behavior surveillance surveys (BSS), condom social marketing issues, and STI case management; policy and socioeconomic impact assessments; and epidemiologic impact modeling. To conduct these types of evaluation at the country program level, especially in the area of behavior surveillance, is not only cost efficient, but also effective in situations where the particular effects of individual projects implemented by different donors cannot be determined.
The priority at the global level should be to develop new evaluation methodologies and to address global issues with large-scale and rigorous research. New methodologies should include development and testing of new indicators and the use of modeling to evaluate the effects of HIV interventions. Using this multilevel approach (subproject, country, and global) to prioritize the degree of rigor needed to evaluate programs and projects has enabled AIDSCAP/FHI to develop programs that can provide data on worldwide and national epidemic trends while also remaining flexible enough to respond to needs at the subproject level.
|
Stage 1 - Awareness
Stage 2 - Knowledge
Stage 3 - Risk Assessment
Stage 4 - Action Indirect
Stage 4 - Action Direct
|
Moving Beyond Prevention Indicators
The original AIDSCAP/FHI evaluation strategy stipulated the measurement of behavioral indicators specific to target groups. In the early stages of the project, evaluation plans called for the use of core measures similar to those being developed at the time by WHO/GPA, USAID, CDC, and AIDSCAP. These PIs were intended to measure program impact at the country program level, using a standardized protocol that would allow for comparisons among and between countries. AIDSCAP/FHI used these indicators, but adapted them for specific target groups. The indicators focused on knowledge of prevention (an early stage of behavior change) and later stages of behavior change related to sexual partner networking (contact with nonregular sex partners) and condom use with high-risk partners.
As the project evolved, however, these initial end-stage indicators were perceived as insufficient for evaluating trends in sexual behavior among various target groups. In response to this insufficiency, AIDSCAP/FHI shifted to gender- and target-group-specific evaluation indicators that represented behavior change as a continuum between knowledge of prevention and actual sexual behavior change, including partner reduction and condom use (see Table 22). These indicators continue to be evaluated.
|
AIDSCAP's final evaluation of its HIV/AIDS program in Cameroon illustrates how data gathered using a variety of evaluation methods can enrich our understanding of the outcomes and impact of prevention efforts. Process data, complementary qualitative and quantitative behavioral data, and a limited amount of biological data were used to assess the outcome and impact of AIDSCAP's activities. In 6 years, the AIDSCAP program in Cameroon, through peer education and community-based outreach, reached more than 180,000 youth, university, and secondary school students, commercial sex workers, military personnel, transport workers, and owners of bars and hotels. Almost 2,000 peer educators were trained to teach their families, friends, neighbors, and coworkers about HIV/AIDS and to refer them for STI treatment and other prevention services. An aggressive social marketing project sold more than 35 million condoms, with monthly sales in 1996 exceeding the total number of condoms sold in 1989, and more than 1 million educational materials, including videos, radio and TV spots, and printed materials, were disseminated. These process data show that prevention activities occurred on a sufficient level to affect behavior. The program focused on changing sexual behavior by promoting abstinence for young adults, fidelity for couples, partner reduction, and condom use. Results of pre-post KABP surveys conducted with members of target audiences showed significant increases in knowledge of HIV/AIDS prevention methods among all groups, and indicated significant decreases in the prevalence of high-risk behavior among most of the groups. One of the program's most important achievements was an increase in the number of persons seeking appropriate treatment for STIs, as STI prevalence is a serious health problem that also contributes to the HIV/AIDS epidemic in Cameroon. The percentage of those reporting that they had sought STI care from a health-care professional rose among university students, military personnel, and commercial sex workers and their clients. For commercial sex workers, there was a dramatic 4-year increase from 32 to 86 percent among. These results suggest that the program's emphasis on improving STI services at health care facilities and referring people to those services was successful. Training to change providers' attitudes toward STI patients was instrumental in improving STI treatment-seeking behavior, according to Dr. Mpoudi Ngolle, the chief of Cameroon's national AIDS control program. "Now everybody knows how well people are treated in the hospital," he said. "And as a result, they won't hesitate to go there." Attitudes toward condom use also changed, as condom use rose among female university students, commercial sex workers and their clients, and military personnel, with a particularly significant increase during commercial sex. The percentage of sex workers that reported ever using a condom rose steadily from 28 percent in 1988 to 88 percent in 1996, and the proportion of clients who had ever used a condom also increased, from 55 percent in 1990 to 81 percent in 1996. Interviews and focus group discussions with commercial sex workers and their clients provided further evidence of a dramatic shift in attitudes toward condoms. "There has certainly been a change in behavior because most of the sex workers today, you will notice that they all use condoms," said one Yaoundé sex worker. "Ten years ago you could not see such a thing in this country. These condoms which have been so decried, so condemned at one time, are now appreciated." Sex workers reported significant increases in consistent condom use, from 52 percent in 1990 to 75 percent in 1996, but only with men who were nonregular clients. Evaluation results suggest that the closer the relationship, the less likely women are to request condom use. About 63 percent say they use condoms consistently with regular clients, and only 13 percent report condom use with their regular, nonpaying partners. Few biological data are available to confirm AIDSCAP's behavioral findings in Cameroon. Sentinel surveillance among women attending antenatal clinics indicates that HIV prevalence is rising in the general population. However, the results of seroprevalence studies conducted between 1992 and 1997 among one of the program's primary target groups -- commercial sex workers in the cities of Yaoundé and Douala -- suggest that infection rates may be stabilizing or even decreasing among sex workers in cities where the use of condoms in commercial sex is relatively high after more than 7 years of comprehensive HIV/AIDS prevention campaigns. |
Triangulation and the Role of Qualitative Research
In practice, AIDSCAP/FHI used nonexperimental observational methods to evaluate behavioral outcomes. It is important to recognize, however, that while a pre-post evaluation design may be useful for assessing a prevention program's proficiency in delivering services, it is not an effective approach for measuring program effectiveness, since the inference of cause and effect from pre-post evaluation data does not take into account alternative explanations for behavior change over time. Behavior change interventions designed to reduce risk behavior must be evaluated and analyzed in such a way that accounts for the social and cultural context within which the intervention program is operating.
The rationale for the AIDSCAP Project's multiple methodology technique for evaluating sexual behavior change is simple: sexual behavior is an extraordinarily difficult area of human behavior to research and understand. Therefore, the use of multiple techniques, or triangulation, to document and interpret reported behaviors helps project managers design better HIV prevention programs. With the multiple technique methodology, findings from qualitative evaluation research and results from quantitative KABP surveys were analyzed together to assess changes in sexual behavior among target groups targeted by AIDSCAP/FHI interventions.
To complement quantitative research methodologies, AIDSCAP used qualitative methodologies to understand the context in which target group-based behavior change is tracked. In a qualitative approach, the contextual interpretation is based on the words of target population members, as direct quotations from transcripts are collected using different qualitative methodologies. Evaluators and implementing agencies working on AIDSCAP/FHI projects used focus group discussions, individual (key informant) interviews, and rapid ethnographic studies to collect qualitative data.
While triangulation is ideal, it poses problems for evaluators' limited budgets and time frames and is further limited by political, social, and cultural realities. Nevertheless, most evaluation experts agree that using qualitative research to complement quantitative data greatly reduces the effect of systemic bias on the data.
Behavioral Surveillance Surveys (BSS)
Quantitative estimates of expected behavioral outcomes require precise assessments of baseline levels and an understanding of how much change is meaningful in the selected intervention settings. Without this knowledge, the tasks of setting sensitive targets for expected levels of change in a pre-post design is difficult.
In response to these limitations, AIDSCAP/FHI moved toward a behavioral surveillance approach, which was first implemented in Bangkok in 1992 and later used in India, Senegal, and Indonesia. Behavioral surveillance is a monitoring and evaluation system designed to track trends in knowledge, attitudes, beliefs, and practices related to HIV prevention in various risk groups within the overall population. Such a system allows for monitoring of decreased risk in some groups and detection of emerging or increasing risk in others. It also helps program managers and stakeholders evaluate program success while reassessing programmatic needs in a changing environment.
AIDSCAP/FHI's behavioral surveillance methodology
- provides more targeted information than systems that collect data on general populations only.
- eliminates the need to collect data separately in a multitude of projects that reach the same target groups.
- ensures comparability and a degree of standardization not necessarily present when data is collected by a variety of different implementing agencies.
- provides data at regular intervals to guide interventions during a project instead of only at the beginning and the end.
- moves focus away from the measurement of unpredictable levels of change toward monitoring trends in various target groups.
- allows evaluators to add or withdraw different target groups from data collection, which ensures an element of flexibility.
Measuring the Potential Impact of HIV/AIDS Interventions
Given the difficulties and high costs associated with direct measurement of the impact of HIV prevention programs through large-scale incidence studies, more emphasis has been placed on developing other methods of assessing impact. These methods involve the use of multiple techniques for examining the relationship between available biological, behavioral, and sociodemographic data. The focus is on establishing linkages between outcome data from program interventions and patterns of HIV prevalence and incidence and on estimating cost-effectiveness.
Assessment methods can be categorized in several ways. Methods include the application of models to estimate the number of HIV infections that were averted because of prevention activities; application and validation of models to estimate HIV incidence rates and prevalence in selected populations; application of methodologies for linking behavioral and biological data; and the expansion of effectiveness analysis to cost-effectiveness analysis.
For example, the newly developed AVERT model is used to estimate the impact of subpopulation-specific behavior changes observed in AIDSCAP/FHI projects on the number of primary HIV transmissions averted. These estimates provide a better understanding of the effect of existing HIV prevention strategies and are also useful to program managers and other stakeholders in their efforts to set priorities for future HIV programming.
|
When AIDSCAP/FHI first began work in Senegal in 1994, HIV/AIDS-related quantitative data were not available to help programmers design appropriate prevention programs. Only one 1989 survey from Dakar and some Demographic and Health Survey (DHS) data from 1992 existed, and they did not include much information useful for HIV/AIDS prevention strategies. Data on indicators related to knowledge about STI and HIV/AIDS, sexual partner networking, and condom use were needed to plan interventions and to serve as a baseline for future evaluations. With the multitude of interventions serving multiple and overlapping target groups in Senegal, however, it was not practical or efficient to gather baseline data for each one of them. In 1995, inspired by the success of the BSS in Thailand and other Asian countries, AIDSCAP/FHI decided to apply the methodology in Senegal, its first introduction into Africa. In Asia, the BSS had proved the best means of tracking trends in sexual behavior among target groups, allowing implementing agencies to better gauge the gaps in understanding and behavior to be addressed by interventions. For Senegal, introducing BSS meant that a large amount of previously unavailable information about HIV/AIDS-related knowledge, attitudes, beliefs, and practices in the population would be collected and used to guide HIV/AIDS prevention and control efforts. In Senegal, as in Asian countries, the first step was to build consensus for the BSS by involving all stakeholders early in the planning stage. The input of the national AIDS control program, AIDSCAP/FHI implementing agencies (IAs), USAID, and other international partners was critical to determining the locations and target groups for BSS and ensuring that the information gathered would be relevant for their programs. At first, some stakeholders were skeptical of the viability of the BSS, but interest slowly grew among the various partners and developed into enthusiastic support as the first round of data collected became available. Among the valuable information that became evident from the first round of the BSS was the following:
The success of the first round of the BSS led the head of the national AIDS control program to promote its expansion to all regions of Senegal and actively encourage other partners and donors to participate in this effort. One important donor, UNICEF, has already solicited a draft proposal from ISADE, the research firm that implemented the first round for AIDSCAP/FHI, to conduct BSS in other regions of the country. |
AIDSCAP's AVERT Model: A Case Study
HIV/AIDS programs typically measure progress by assessing changes in behavior among target audiences. But financial, logistical, and technical constraints usually make it impossible for them to answer the most important question about a prevention intervention: did the reported behavior change lead to reductions in HIV transmission?
AIDSCAP's AVERT model is an excellent tool for answering that question. This computer model was designed to estimate the number of infections averted through behavior changes resulting from prevention efforts.
AIDSCAP used AVERT to gain a better understanding of the impact of one of the first pilot studies of targeted periodic presumptive STI treatment in the developing world. Such treatment has been proposed as an option for reducing STIs in groups at high risk of infection -- particularly high-risk women -- who often do not experience STI symptoms and may not seek treatment otherwise.
The study offered free monthly examinations, treatment and counseling, and community-based peer education on STI/HIV prevention, to women engaged in commercial sex and others at high risk of STIs in a South African mining community where migrant employees live away from their families for much of the year. All the women who used the services were treated for the most prevalent STIs in the area with a single-dose antibiotic.
Study results showed that this approach was effective, resulting in dramatic decreases in STI prevalence among the women using the service and their miner partners After just 9 months of intervention, overall STI prevalence had dropped by 30 percent among the commercial sex workers and by 20 percent among their miner clientele.
Since prompt, effective STI treatment and peer education are key HIV/AIDS prevention strategies, the researchers and the mining company managers were interested in estimating the impact these interventions might have had on HIV transmission. Estimates produced by the AVERT model showed them just how powerful an HIV intervention presumptive STI treatment could be in such a high-risk environment.
The model incorporates the most current research on the probability of HIV transmission under different conditions, such as the presence or absence of STI. By modeling pre-post intervention scenarios of high-risk behavior among pairs of target populations, AVERT can produce estimates of the subsequent difference in new HIV infections.
For the analysis of the pilot study in South Africa, AIDSCAP researchers constructed pre-post intervention scenarios based on reported behavior and STI test results. These scenarios included the average number of sexual partners and sexual contacts per partner, overall prevalence of ulcerative and nonulcerative STIs, and condom use. They assumed that the 400 women who had regularly used the STI treatment and counseling services had had sexual contact with 4,000 miners living in the nearby hostels -- an assumption based on the conservative estimate that only 40 percent of the miners were engaging in commercial sex.
After 9 months, it was estimated that the overall prevalence of genital ulcer disease (GUD) had dropped by 30 percent and nonulcerative STI rates had fallen by 32 percent. The women had reduced the number of clients by 20 percent, and reported condom use by the clients had increased from 13 to 19 percent. Modeling these scenarios, AVERT estimated that the intervention had averted a total of 237 new HIV infections for the year -- 40 among the women and 195 among the miners.
The model was also used to project the potential impact of continuing the intervention. It showed that if the project goals of 50 percent condom use in commercial sex and an 80 percent reduction in STI rates were achieved during the next 2 to 3 years, the estimated annual cumulative incidence of HIV would decline from 52 to 12 percent among the women and from 13 to 29 percent among their miner clientele.
AVERT estimates enabled the researchers to do a cost-benefit analysis showing that for every dollar spent on presumptive treatment and peer education, the mining company had saved more than eight dollars in treatment costs for HIV-related illnesses among its employees. This conclusion persuaded the Harmony Mine management to continue and expand the intervention.
| Assumptions | Scenario 1 | Scenario 2 | Results | |
| avg. annual partners (women) | 40 | 32 | . | |
| avg. annual contacts per partner (women) | 10 | 10 | ||
| avg. annual contacts (miners) | 4 | 3.2 | ||
| avg. annual contacts per partner (miners) | 10 | 10 | ||
| GUD prevalence | 10% | 7% | ||
| non-GUD prevalence | 25% | 17% | ||
| condom use | 13% | 29% | ||
| Results | Difference | Percent | ||
| probable HIV infections (women) | 103 | 62 | 41 | -40% |
| probable HIV infections (men) | 405 | 209 | 196 | -48% |
Challenges for the Future
During the 6-year project, AIDSCAP/FHI designed, implemented, and developed evaluation research methodologies capable of bringing qualitative and quantitative data together to help assess the efficacy and cost-efficiency of HIV/AIDS prevention and control programs. The challenges of evaluation and assessment, however, will continue in the next phase of USAID's worldwide HIV/AIDS prevention and control interventions.
Efficacy of Intervention Strategies
There remains a need for a limited number of well-designed efficacy trials of existing intervention strategies, especially behavioral interventions designed to reduce the sexual transmission of HIV and other STIs. These studies must be of sufficient size to yield clear results and should be designed to allow inferences about cause-effect relationships.
Contextual Analysis: Linking Behavioral and Biological Data
Program evaluation is intrinsically complex due to the temporal evolution of epidemics and our poor understanding of how different behaviors and epidemiological factors influence epidemic patterns as they move from an epidemic phase to an endemic state. Changes in HIV prevalence may be indicative of the long-term impact of multiple HIV/AIDS prevention interventions, but it is very difficult to prove that observed decreases in prevalence trends are the result of HIV prevention programs. Other factors such as mortality, migration, and saturation of the population at risk can also account for such changes.
There is an emerging consensus among evaluation experts that prevention programs need to investigate trends in infection alongside trends in behavior that may lead to that infection. To this end, HIV/STI serosurveillance data have to be collected in conjunction with behavioral, socioeconomic, and sociodemographic data. The combined analysis of these sets of data will provide the necessary context and range of information for an interpretation and explanation of the epidemiological data collected by sentinel serosurveillance surveys.
Data Quality
In the next phase, USAID-supported prevention and control programs need to focus on improving the quality of collected data. Because AIDSCAP/FHI's commitment to capacity building resulted in the involvement of local organizations and researchers in collecting evaluation data, the quality of the data was expected to vary among the different countries. In addition, comparing data from studies and making general conclusions are difficult when studies use different outcome measures or different wordings in survey instruments.
Data collection systems require substantial attention and maintenance to ensure the integrity of the data collected. Besides local capacity building for the collection, analysis, and dissemination of evaluation data, an additional challenge for prevention programs should be the identification of implementing partners dedicated to the quality of evaluation results. Planning evaluation in a participatory fashion is essential for achieving the delicate balance between practical needs and methodological desirability. Active participation of key stakeholders who have a vested interest in the quality and reliability of the results produced by data collection systems is probably the single most important factor for ensuring that data produced by these efforts will be reliable, of good quality, relevant, and timely.
Self-Reported Behavior
The validity of survey data on sexual behavior is very difficult to establish because a limited range of evidence can be collected to provide independent corroboration of the validity of self-reported behaviors. However, it should be emphasized that the quality of data also depends on the level of detail of information elicited by the survey instrument. For example, it is not sufficient to know whether respondents have ever used condoms or whether they have begun to use them. It is also important to determine the frequency of use and to ascertain the conditions that influenced use. This example underscores the need to collect data that are not only valid and reliable but also meaningful.
Emphasis on Sustained Behavior Change
HIV/AIDS prevention programs operating for some years may experience increasing difficulties in detecting changes in outcome variables since the interventions have reinforcing rather than new effects. As a result, the size of potential program effects is becoming smaller, and the sample sizes necessary to measure these effects will increase accordingly. Maintenance of reported behavior change should receive greater emphasis in future evaluations.
Cross-Cutting Issues
Evaluation systems should address crosscutting issues. In particular, evaluation research should emphasize improved approaches to evaluating capacity building, policy and gender initiatives, social marketing concepts, and alternative nonclinical care strategies.
