Evaluating HIV/AIDS prevention and care programs is a never-ending challenge, but recognizing its importance in improving current interventions may help to enhance the success of future initiatives.

There are probably as many definitions of "program evaluation" or "evaluation research" as there are program evaluators. Our approach to evaluating HIV/AIDS prevention and care programs is best captured by a description of evaluation provided by Michael Quinn Patton1:

"I use the term evaluation quite broadly to include any effort to increase human effectiveness through systematic data-based inquiry. When one examines and judges accomplishments and effectiveness, one is engaged in evaluation. When this examination of effectiveness is conducted systematically and empirically through careful data collection and thoughtful analysis, one is engaged in evaluation research…. Evaluation is applied research, or a type of "action science." This distinguishes evaluation research from basic academic research….The purpose of applied research and evaluation is to inform action, enhance decision-making, and apply knowledge to solve human and societal problems….Applied evaluative research is judged by its usefulness in making human actions and interventions more effective and by its practical utility to decision makers, policymakers and others who have a stake in efforts to improve the world."

Such an evaluation approach is utilization-focused. This approach emphasizes the interests of key stakeholders and primary users of the information at all levels, for example the donor, the host country, and the implementing agencies. It applies socio-epidemiological research to identify ways to improve the design and implementation of HIV/AIDS prevention and care programs.

This chapter first describes several considerations that are fundamental to planning an evaluation effort. It then presents a comprehensive framework for country programs by explaining the major types of evaluation and discussing several important issues related to planning evaluation programs and improving their ability to measure program effects.

Basic Considerations

Several considerations underlie the decisionmaking process about HIV/AIDS program evaluation. The selection of an appropriate evaluation concept for an AIDS prevention program is crucial because it determines the guiding philosophy behind the actual evaluation process. A number of theorists and evaluation practitioners have proposed various conceptual approaches to evaluation. These approaches differ in their conception as to what evaluation is, what the relationship with the primary client and other stakeholders should be, who should be making the relevant value judgments regarding the program, and the criteria for judging the evaluation process itself.

The conceptual approach debate was, and is for the most part, a debate about the best ways to measure and interpret change. It has highlighted a series of methodological dimensions among which there are variations in emphasis. These dimensions focus attention on some of the options available for making decisions about methods. Today, for example, there is consensus that both quantitative and qualitative data are valued and recognized as legitimate for program evaluation. In fact, these methods are by no means incompatible and should be used in combination2.

Deciding what and how much data to gather in an evaluation involves difficult methodological decisions and trade-offs between the quality and utility of information. An evaluation approach that uses multiple data collection methods, both quantitative and qualitative, is more likely to address diverse evaluation needs than is a more limited approach. At the same time, research priorities must be sensitive to competing needs for resources in an environment in which the HIV/AIDS epidemic is growing rapidly and evaluation is sometimes considered a luxury. It is a major task of the evaluator to match research methods to the reality of particular evaluation questions and to the available resources.

There is also a need for evaluation researchers to play an active role, not merely a consultative one, in making design decisions for program assessments. Although a program evaluator should be a neutral scientific observer, he or she can also mediate between different stakeholder groups, can enable others through a participatory evaluation approach, and can advocate for the dissemination of evaluation results within the larger arena of decisionmaking3. Planning evaluation and data collection activities in a participatory fashion is essential for achieving the delicate balance between practical needs and methodological desirability. Key stakeholders should be included in the planning process and every effort should be made for effective use of limited resources. Ensuring active support and participation of key stakeholders who have an interest in the results obtained by various data collection systems is particularly important for programs funded by external donors that use host country institutions for data collection activities. Data produced by these efforts will have a better chance to be timely and of acceptable quality. Whenever possible, participants, including implementing institutions, host-country collaborators and local representatives of donor agencies, should attempt to reach consensus regarding data needs.

A Comprehensive Evaluation Framework For Country Programs

HIV/AIDS prevention and care programs need to be evaluated at different phases of the program cycle. Table 1-1 outlines a framework for comprehensive program evaluation. All stages of evaluation have to be considered together to provide an overall picture of the program because no single data collection approach can supply all the information necessary to improve program performance or affect policy change. Multiple complementary evaluation approaches and multiple methodologies (qualitative and quantitative) have to be applied to address different evaluation needs.

Formative Evaluation
Formative evaluation should be conducted during the planning (or replanning) stage of a prevention and care program to identify and resolve intervention and evaluation issues before the program is widely implemented. This is the time when flexibility is greatest and program sponsors are freer to make decisions about how to proceed.

Formative evaluation explores the need for interventions, provides the information necessary to define realistic goals and objectives for the program interventions, and helps program planners make tentative decisions about effective, feasible intervention strategies and how to carry them out. Formative evaluation can also be used as an exploratory tool as the project is being carried out to provide feedback to project managers to help them adjust program objectives to changing situations. Formative evaluation research can identify unacceptable or ineffective intervention approaches, designs, and concepts.

Because of the urgency of the HIV/AIDS problem, many prevention programs have rushed to carry out interventions without preparing first by conducting thoughtful formative evaluation. The lack of this type of evaluation is particularly felt in community-based interventions designed to reduce sexual transmission of HIV. In many cases, interventions have been based on ideas developed outside of the context of the lives of the people to whom the interventions have been delivered. The literature on behavioral change interventions is full of examples of ideas that made perfect sense in the abstract but failed completely in the "real world," mainly because the ideas were unacceptable to the target audience or were not stated in ways that were relevant to the lives of those people. A fuller understanding of the issues might well have led planners to redesign the intervention to make it more appealing to the selected audience. Fortunately, this situation is changing because formative evaluation is now being applied more frequently in designing prevention programs.

Formative evaluations use a mix of research methods that can rapidly provide relevant information to program designers. These methods include:

The most frequently cited methodological criticism of formative evaluation is its lack of external validity or generalizability. Because the results of the evaluation derive from small-scale rapid assessment procedures and/or pilot studies, one cannot generalize from them to a larger population. Despite this limitation, formative evaluation research can usually identify unacceptable or ineffective intervention approaches, designs, and concepts. However, even with adequate formative evaluation at the program planning stage, there is no guarantee that a prevention program will be effective when finally implemented; it may not be implemented adequately enough to be effective.

Process Evaluation
Once activities are underway, there is a need to examine whether they are being carried out correctly, on time, and within budget. Process evaluation addresses such basic questions as,

"To what extent are planned intervention activities actually realized?" and "What services are provided, to whom, when, how often, for how long, and in what context?" Both input (the basic resources required in terms of manpower, money, material, and time) and output (the immediate service improvement expressed as distributed commodities, trained staff, and service units delivered) are key elements of process evaluation. These questions are often answered in quantitative terms. Qualitative evidence of how and why a prevention program works or fails to work is equally important in answering process evaluation questions. Process evaluation requires getting close to data, becoming intimately acquainted with the details of the program, and observing not only anticipated effects but also unanticipated consequences. An understanding of the processes through which intervention activities achieve effects can help to explain the outcome of the intervention. Process evaluation, however, does not demonstrate whether interventions are effective.

Process evaluation can also play an important role in improving or modifying interventions by providing the information necessary to adjust delivery strategies or program objectives in a changing epidemic. Process-oriented evaluation is carried out throughout the course of the program implementation and should use different methodological approaches to assess service delivery, ranging from reviews of service records and regular reporting systems, key informant interviews, exit interviews of service users, direct observations by 'mystery clients' (for example, in sexually transmitted infection [STI] and voluntary counseling and testing [VCT] services) to quantitative population-based surveys to assess program coverage and barriers to service use. Different qualitative and quantitative study designs that are complementary to one another provide together the most comprehensive information. (See Section II—Chapters 3 through 7—for details on these approaches.)

Effectiveness Evaluation: Assessing Outcome and Impact
Evaluating the effectiveness of AIDS prevention programs will almost always require quantitative measurements. These measurements will assess the extent to which the objectives of the program were achieved. Effectiveness evaluation is used to answer the questions, "What outcomes were observed?" "What do the outcomes mean?," and "Does the program make a difference?"

Taking into account the various implementation stages of HIV/AIDS prevention programs and the fact that, over time, new age cohorts become sexually active, it is advisable to stratify effectiveness evaluation by short-term and intermediate program effects (program outcome) and long-term program effects (program impact). Table 1-2 provides examples of program outcome and impact measures for these different stages. Changes in HIV/AIDS-related attitudes, the reduction of risk behaviors and adoption of protective behaviors, and changes in STI rates are considered to be the most appropriate short-term or intermediate (also called proximate) outcome measures for interventions designed to reduce sexual transmission of HIV. Long-term effects include impact on HIV/AIDS trends, sustainability issues, and improved societal response.

Outcome and impact evaluation is intimately connected with process evaluation. Process information can help the evaluator to understand how and why interventions have achieved their effects and, perhaps, what is actually making the difference. Examining outcome/impact indicators without assessing the process of program implementation could lead to erroneous conclusions regarding the effectiveness of the intervention.

Program goals and objectives have to be carefully defined to allow the selection of appropriate outcome and impact measures to assess the effectiveness of an AIDS prevention program. Effectiveness evaluation is generally based on indicators that provide quantitative value from which the outcome and impact of interventions can be measured. Because multiple interventions working synergistically together are most effective in producing behavior change, surveys should not be typically designed to capture the effects of one single intervention (see Figure 1-1 later in this chapter and Box 2-2 in Chapter 2). Rather, they should be designed to measure behavioral trends in population groups who are exposed to combined interventions. The evaluation of one intervention is usually conducted through rigorous and expensive controlled trials.

Cost-effectiveness Analysis
Cost-effectiveness analysis also measures program effectiveness, but expands the analysis by adding a measure of program cost per unit of effect (for example, per number of HIV infections averted). By comparing the costs and consequences of various interventions, cost analyses and cost effectiveness estimates can assist in priority setting, resource allocation decisions, and program design. (Chapter 17, "Guidelines for Performing Cost-effectiveness Analysis of HIV/AIDS Prevention and Care Programs" provides more detail on this type of evaluation.)

The Attribution Dilemma: Are Observed Changes A Result Of Prevention Interventions?

The ultimate goal of any HIV prevention program is to reduce the number of new infections. Program evaluation is intrinsically complex, however, due to the temporal evolution of epidemics and our poor understanding of how different behaviors and epidemiologic factors influence epidemic patterns as they move from an epidemic phase to an endemic state. Several factors unrelated to intervention effects can contribute to the observed stabilization or decreases in the prevalence or incidence of HIV in a given setting. They include:

Determining whether observed changes in HIV incidence and prevalence are a reflection of the natural history of the epidemic or due to intervention effects is a critical evaluation issue. This is particularly true when evaluating behavior changes in the face of growing numbers of people with AIDS-related illnesses because there is evidence that secular trends toward risk reduction will occur. For example, having a friend or relative with HIV/AIDS may influence adolescents to delay the onset of sexual relations or motivate those with non-regular sex partners to use condoms. Human sexual behavior is influenced and shaped by many factors and exposure to an HIV prevention program is only one of them.

The question of whether behavior changes can be attributed to prevention programs, especially in countries with advanced HIV epidemics, has created some friction between stakeholders and program implementers at the field level. Their different perspectives on this issue also reflect fundamental differences regarding the criteria for judging the process of program evaluation itself. From a public health perspective, it may not matter whether the observed changes are due to a particular intervention. What is most important is that sexual practices have become safer and HIV infection should subsequently decrease.

From the cost-effectiveness or policy perspective, however, it is important to determine what caused the observed changes in sexual behavior. If the changes would have occurred without a particular intervention that was designed to contribute to the observed changes, the costs

of the intervention could be considered as resources better spent on something more useful4. Prevention programs are under growing pressure to estimate which approaches work best for specific target populations in different epidemiologic settings with a given level of inputs in order to allocate resources in a cost-effective manner5. Effectiveness evaluation, therefore, is critical because it can answer a basic question, "Does the program make a difference?"

A vexing task of assessing program effectiveness is to disentangle the attributable affects of a prevention program from the gross outcome and impact observed. Such estimates can be made with varying degrees of plausibility, but not with certainty. A general principle applies here: The more rigorous the research design, the more convincing the resulting estimate6. A hierarchy of evidence based on the study design can be established that reflects the degree of certainty in concluding that a given proportion of the observed changes in behavior is attributable to the intervention program and is not the result of other factors7.

Table 1-3 presents a ranking of different study designs according to their decreasing strength of evidence. Non-experimental observational methods with no control groups have been routinely used in behavioral outcome evaluations. It is important to recognize, however, that a before-and-after evaluation design with no comparison groups may be useful for assessing a prevention program's proficiency in delivering services, but it is not a very convincing design to measure program effectiveness. The inference of cause and effect from such a design is problematic because competing explanations for across-time behavioral changes cannot be ruled out8.

In some situations, the evaluation could assess "exposure" to an intervention program or a specific element, and determine the extent of the association of that exposure with the desired outcome. This method, however, can be limited in its utility by factors such as lack of association of services or products with the intervention and inaccurate reporting by respondents of their participation in the intervention.

The interpretation of program evaluation data should always be approached with caution. In most situations, the program and evaluation process as a whole is not a rigorously controlled experimental trial. The ability of an evaluation to precisely determine the true extent of a program's effectiveness is often limited by time, resources, and the lack of a rigorous design. Many factors can confuse or confound the results measured, and biases can be introduced by a range of factors inherent to the problem of HIV/AIDS, the available measurement options, and those conducting the evaluation. One of the most difficult questions to answer in any evaluation is that of attributing any measured effect to the program being evaluated. Defining the web of interacting and overlapping influences is extremely difficult, and is one of the reasons why so many programs have difficulty attributing results to their actions. At some point, we need to stop worrying about attribution in such settings and focus on monitoring the changes as they occur.

The Role Of Triangulation

Triangulation can be achieved through using multiple data sources, different researchers, multiple perspectives to interpret a single set of data, or multiple methods applied to a single program, problem, or issue. In the absence of rigorous controlled trials, data triangulation procedures must be applied to substantiate a link between interventions and observed behavior changes. For example, process evaluation data on condom sales, the intensity of peer education, or the quality and coverage of media campaigns can be combined with an analysis of behavioral outcome data to provide an understanding of the process through which an intervention has achieved its effects. Results from behavioral surveys should be analyzed together with findings from qualitative evaluation research that is carried out in sub-samples of surveyed target populations. Such research can include focus group discussions, key informant interviews, and rapid ethnographic studies. This type of analysis will allow a more appropriate interpretation of observed outcome data because they are the likely results of the aggregate effects of multiple interventions as well as environmental and personal factors.

Many of the areas that need to be measured to evaluate HIV/AIDS programs are sensitive and very personal in nature, such as sexual behaviors or personal attitudes toward persons with HIV/AIDS. Validity and reliability are critical issues for sexual behavior research because the behaviors cannot be directly observed. Self-reports of sexual behaviors in the absence of additional evidence are often considered invalid and unreliable by stakeholders for whom such data are sensitive and run against firmly held cultural norms.

One of the best methods for promoting reliability and validity, therefore, is to triangulate behavioral data with all other available and relevant biological, behavioral, and process data to explain more comprehensively the context in which risk behaviors take place. (Chapter 11, "Assessing the Validity and Reliability of Self-reported Behavioral Data," provides more detail on this issue.)

Given the abundance of AIDS-related research conducted in many countries, secondary data are a source of material for triangulation. Multiple-method triangulation is probably the most common triangulation technique. Rapid ethnographic research, combining semistructured information gathering with mapping, participant observation, and in-depth interviews, is another possibility. Focus group discussions have been widely used, as have individual in-depth interviews or key informant interviews, to obtain stakeholders' (or other key individuals') opinions about target groups' behavior. (Chapter 12, "The Role of Qualitative Data in Evaluating HIV Programs," provides more information on qualitative evaluation tools.)

Finally, it is important to realize that behavior change interventions have to be in place for sufficient amounts of time and on a large enough scale to have an impact on personal behavior, social norms in communities, and ultimately on the epidemic. The example of Thailand shows that a focused intervention strategy implemented on a national scale can result in substantial declines in HIV incidence and prevalence in targeted populations. It is also an example of applied triangulation of data: STI/HIV trends were systematically collected by sentinel surveillance systems, and behavioral surveillance data provided the necessary supplementary information to interpret the observed epidemiological trends. There is now growing consensus that country programs need to monitor risk behavior trends together with trends in HIV infection. (See Chapter 8, "Uses of Behavioral Data for Program Evaluation" for more on this issue.)

Choice Of Indicators

One of the critical steps in designing and carrying out an evaluation of an HIV/AIDS program, or any program for that matter, is selecting appropriate indicators. This can be a fairly straightforward process if the objectives of the program have been clearly stated and presented in terms that define quantity, quality, and time frame of a particular aspect of the program. Even with well-defined objectives, however, the choice of indicators for the evaluation of many programs requires careful thought and consideration of both theoretical and practical elements.

The following questions can be helpful in selecting indicators:

Selecting indicators and setting targets is usually done during the process of program planning and replanning, preferably in a participatory way with the implementing agency and key stakeholders. Setting targets and benchmarks should also include information from similar types of interventions, so that the targets set are realistic from the perspective of the target population, resource allocation, and intervention type.

While the level of attainment to be measured by the indicator is not actually part of the indicator itself, it is a critical factor. The magnitude of the level to be measured affects the size of the sample of the population needed to estimate that level accurately. It may also help evaluators select additional or supplemental indicators that might assist in later interpretations of the results.

Ideally, indicators should be:

Validity is inherent in the actual content of the indicator and also depends on its potential for being measured. Reliability is inherent in the methodology used to measure the indicator and in the person using the methodology. Many familiar outcome indicators in HIV/AIDS prevention, such as measures of condom use, provide challenges to the evaluator with respect to validity and reliability.

Interpreting outcome indicators for behavioral interventions that promote safer sex is further complicated by the fact that risk behaviors are measured in relative terms. For example, percentage figures of condom use measure the proportion of sexual exposures that are considered to be safe. These may or may not reflect the absolute number of sex acts that place individuals at risk for exposure to sexual transmission. Ten percent condom use in 10 HIV-associated sexual episodes is still "safer" than 75 percent condom use in 100 HIV-associated sex episodes (9 versus 25 unprotected HIV-associated sex acts, respectively). Therefore, in this example it also would be important to determine the frequency of condom use in absolute terms in a given risk situation. Behavioral surveys have begun to address this dilemma by collecting additional data on "always or consistent" condom use in the context of sexual episodes with non-regular partners.

Table 1-4 lists possible indicators related to different levels of program evaluation. The advantage of relating indicators to specific evaluation levels is that it also helps to identify opportunities for triangulating data. For example, survey data on condom use can be compared with information on condom distribution and availability in a defined intervention area. Or, available data on incident STIs, such as gonorrhea, in the surveyed population could be correlated with the condom data.

In collaboration with national and international partners, the United Nations AIDS Programme and the World Health Organization (UNAIDS/WHO)11 have developed a standard set of indicators for country programs that will refine and expand the prevention indicators (PI) developed by WHO's former Global Program on AIDS (GPA). Moreover, because HIV/AIDS/STI prevention and care programs are affected by many factors, including political commitment, available resources, and the socio-cultural and economic context, a new approach is currently being developed to capture the overall effort of national HIV/AIDS programs. The AIDS Program Effort Index (API) is a composite score comprised of the main components of an effective national response.

The potential advantage of this instrument is that it may yield useful information on the above issues even in the absence of more rigorous monitoring and evaluation systems. Using the key informant assessment approach, it also allows an assessment of areas that are difficult to capture with more objectively measurable indicators (e.g., political support and commitment). However, major concerns have been expressed with regard to the subjectivity and reliability of the API approach. The score depends entirely on the choice of informants, and the informants are likely to change from year to year. Questions have also been raised about the utility of a single composite score in which improvements in some areas may be masked by deterioration in other areas11.

Differentiating Evaluation Efforts

Because of the various constraints on time, available funds, and trained staff, program managers and evaluation planners must balance what is ideal or preferred against what is feasible, useful, relevant, and essential when choosing how to evaluate a particular intervention or program. A useful approach for differentiating evaluation efforts is to define them in three different dimensions: the individual project dimension, the country program dimension, and the international dimension. Using this multi-dimensional approach (individual, country, and international project) to set priorities for the degree of rigor needed to evaluate programs and projects may alleviate some of the tension that arises when universal, standardized evaluation practices conflict with the objectives of individual projects.

One can think of the individual project dimension as an area of service delivery that, in most cases, does not require a rigorous research design to judge its proficiency, unless it is piloting a new intervention or responding to an unanswered research question, such as would occur with a demonstration project. Individual projects carrying out standard intervention strategies that have already been shown to be effective in other similar settings should focus their evaluation activities on formative evaluation (when needed for project planing), process evaluation, and capacity building assessment. Figure 1-1 shows the number of projects in relation to the different levels of evaluation efforts and reflects the current situation in program evaluation. The monitoring and evaluation "pipeline" illustrates that there is usually a reduced number of projects that actually warrant the evaluation of the effectiveness of their implemented prevention activities.

Within the dimension of a country program, several categories of evaluation should be emphasized—intervention outcomes, socioeconomic impact, and changes in societal norms. The guiding principle here is that in a situation in which multiple donors are conducting multiple interventions with overlapping target groups, certain types of evaluation are not appropriate for the scope of an individual project, but rather, should be coordinated and conducted by country or regional programs. Use of such an evaluation approach, especially in the area of behavioral surveys, not only saves money, but also makes sense in environments where the effects of individual projects from different donors cannot be sorted out anyway.

Country program evaluation includes (but is not limited to) the analysis of behavioral trends in different population groups in conjunction with an analysis of HIV/STI surveillance data; the evaluation of social marketing activities related to condoms, drugs, and services; STI case management; scoring the overall effort of the national program (for example, through the AIDS Program Effort Index); socioeconomic impact assessments; and epidemiological modeling of the country's epidemic. Countries will have different programs of evaluation activities, reflecting their different information needs, which are determined by the stage of their epidemic, as well as their political and social environment, existing capacity for research, and available financial resources.

Evaluation efforts on the international dimension may address still existing uncertainties about which set of prevention interventions works best, in which setting, for whom, and under what circumstances (emphasis on cost-effectiveness analysis). This type of evaluation, however, requires large-scale community-based controlled trials that are certainly beyond the scope of individual projects or even national programs.

Given the difficulties and high costs associated with directly measuring the impact of HIV prevention programs through large-scale incidence studies, more emphasis has now been placed on developing other methods of assessing impacts, for example through modeling. (Chapter 15, "Translating Survey Data into Program Impact: the AVERT Model," provides more detail on one such effort.)

Conclusion

HIV/AIDS prevention and care program evaluation is applied socio-epidemiological research whose main purpose is to identify and solve practical problems and guide program managers and planners in improving the design and implementation of prevention and care activities. This perspective not only determines the role of program evaluation but also how an evaluation should be conducted, including the choice of indicators and levels of efforts in a given setting.

By applying different methods from several disciplines to many types of problems, program evaluation is a comprehensive research approach committed to meeting the needs of stakeholder groups as well as the requirements of the scientific community. This Handbook is designed to assist program evaluators in this critically important task.

Although program evaluation is context specific, a comprehensive framework as outlined in this chapter is helpful in defining the questions that are to be answered by the different types of evaluation during the program cycle. We advocate for a utilization-focused evaluation approach that emphasizes the interests of stakeholders as the primary intended information users. To achieve the delicate balance between practical needs and methodological desirability, it is therefore essential that program evaluations are planned in a participatory fashion with key stakeholders.

Decisionmaking is a political process and program evaluators can play a major role in this process when evaluation efforts are expected to provide information of policymaking significance and relevance. Although evaluation researchers should be neutral scientific observers, there is also a need for them to assume a more active role and, if necessary, mediate between stakeholders with different and sometimes conflicting interests, perspectives, and information needs.

Given the limited resources in most developing countries, assessing the effectiveness of HIV prevention programs will often depart from scientifically ideal designs. In the absence of more rigorous evaluation designs, we urge program managers and evaluators to apply triangulation procedures using multiple complementary methods as well as different data sources. Such a triangulated analysis will provide information comprehensive enough to allow a plausible and valid interpretation of observed outcome data, such as changes in risk behaviors, because they are the likely results of the aggregate effects of multiple interventions as well as environmental and personal factors.

References

1. Patton MQ, editor. Qualitative evaluation and research methods. 2nd edition. Newbury Park (CA): Sage Publications; 1990. p. 11-12.
2. Patton MQ, editor. Utilization-focused evaluation. 2nd edition. Beverly Hills (CA): Sage Publications; 1986. p. 177-217.
3. Thompson JC. Program evaluation within a health promotion framework. Can J Public Health 1992;83(Suppl 1):S67-S71.
4. Coyle S, Boruch R, Turner C, editors. Evaluating AIDSprevention programs. Washington (DC): National Academy Press; 1991. p. 50-82.
5. Mertens TE, Caraël M. Evaluation of HIV/STD prevention, care and support: an update on WHO's approaches. AIDS Educ Prev 1997;9(2):133-145.
6. Rossi PH, Freeman HE, editors. Evaluation: a systematic approach. 5th edition. Newbury Park (CA): Sage Publications; 1993. p. 215-259.
7. Susser M. Some principles in study design for preventing HIV transmission: rigor or reality. Am J Public Health 1996;86(12):1713-1716.
8. Cook TD, Campbell DT, editors. Quasi-experimentation: design and analysis issues for field settings. Chicago: Rand McNally; 1979. p. 95-146.
9. Boerma T, Schwartländer B, Mertens T. A proximate determinants framework for monitoring and evaluation of HIV/AIDS/STD programs. From: Towards Improved Monitoring and Evaluation of HIV Prevention, AIDS Care and STD Control Programs. A joint UNAIDS, WHO, USAID/MEASURE evaluation workshop, Nairobi (Kenya); November 17-20, 1998.
10. WHO/GPA/TCO/SEF/94.1. Evaluation of a national AIDS programme: a methods package. Geneva: World Health Organization; 1994.
11. UNAIDS/00.17E. National AIDS programmes: a guide to monitoring and evaluation. Geneva: Joint United Nations Programme on HIV/AIDS (UNAIDS); 2000.