This chapter provides guidelines for conducting various types of effectiveness analysis. Here we build upon the analysis of costs presented in the Chapter 16, "Guidelines for Assessing the Economic and Financial Costs of HIV/AIDS Prevention and Care Programs," by looking at how cost data can be combined with measures of effectiveness to generate powerful estimates of the value of an intervention.

It is important to remember, however, that the cost-effectiveness of an intervention is only one aspect of a successful intervention program. Some projects have societal goals that are not easily captured through cost-effectiveness analysis alone, and many projects have long-term benefits that are not readily assessed through cost-effectiveness analysis (such as programs with youth).

The first section of this chapter provides a general overview of the types of available effectiveness analysis, and walks the reader though a process of identifying what data are available, and what data are needed for different types of analyses. The types of analysis reviewed include:

• cost analysis;
• effectiveness analysis;
• effectiveness modeling; and
• cost-effectiveness analysis.

The second section of this chapter provides the reader with general guidelines to achieve the following basic steps in developing a cost-effectiveness analysis1:

• framing the problem;
• identifying options to be compared;
• identifying the outcome measure(s);
• identifying intervention and outcome costs (reviewed in the previous chapter);
• constructing a decision tree;
• conducting a sensitivity analysis; and
• presenting the results.

Definitions And Types Of Analyses

There are various approaches to examining the effectiveness of an HIV intervention, and the choice of analytic method depends on both the goals of the analysis and the data that are available. This section reviews various types of possible analyses. It is designed to provide the reader with an overview of the types of analyses that can be conducted, the strengths and limitations of the approaches, and the data needs for each type of analysis. We begin with cost analysis, as this is a required component of cost-effectiveness analysis, and because it has its own unique utility.

Cost Analysis
A cost analysis entails compiling and standardizing the costs of an intervention in order to assess the overall cost, annualized cost, and cost per client. Analyses are often broken out by distinct components of the intervention, by target groups, and by geographic areas of implementation. Costs are also frequently broken out into distinct components of the overall costs, such as recurrent (rent or labor), commodities (condoms or test-kits), and start-up expenses (office furniture or computers).

Cost analyses can take different perspectives, such as the cost to the donor, the non-governmental organization (NGO), or the client. Client costs often include assessments of the willingness to pay for the service, or natural experiments such as the impact of user fees on service use. Another issue to consider in this sort of analysis is the efficiency of the intervention. For example, does the size of the staff reasonably match the flow of clients through the service?

It may be necessary to have a large staff to accommodate busy days, and thus the intervention program can be quite inefficient if staff are idle most of time and very busy at other times. Likewise, the staff may always be busy, but the quality of the service may be poor due to long wait periods resulting in high opportunity costs to the clients (or lost time that could have been spent productively on other activities).

Data required for a cost analysis are described in the previous chapter but, at a minimum, include the line-item costs for the intervention, independent of research costs. It is also helpful to have data on the relative costs of providing an intervention to specific sub-populations, as well as information on seasonal trends in expenditures. Data can come from project budgets, from worksheets completed by project managers, and from interviews with project and donor staff.

It is also important to standardize cost data so that they can be compared across geographic settings and over time. For example, it is usually necessary to convert costs to a common currency metric (such as the dollar) and to discount the expenditures to a standard value of the currency in time to account for inflation and deflation, and variations in currency conversion rates. Moreover, it is also important to try and account for the purchasing power of the currency in the local setting. This is achieved by calculating a cost of living index based upon how much it costs at a given time to purchase a predetermined set of commodities and services in the local setting. The "market-basket" selected for this exercise needs to be a set of common items that are frequently used in all intervention settings. The Purchasing Power Parity (PPP) is a standard scale of the value of currencies, and is available at the World Bank website [www.worldbank.org/html/prddr/trans/m&a96/art7.htm]. The PPP is an estimate of the value of a "market-basket" of good and services based on the relative value of currency across countries over time. Although cost analysis is often conducted many years after an intervention is completed, data collection must be initiated at the time of the intervention.

One problem in compiling the cost data is that once projects disband, it can be difficult to find the original project managers to identify actual intervention costs. In addition, budgets are notorious for not reflecting actual costs. If a client perspective is taken, data are needed on the amount of time, wages, transport and childcare costs, and opportunities lost. Although these costs can be assessed through interviews with clients, the literature does show that people tend to overestimate them in most surveys. Finally, start-up costs should be annuitized over the life of the project; for commodities, an annuity function is normally applied to the items to determine the annual cost. Simple formulas exist to do this. A good source of information on these procedures and associated formulas is Haddix and colleagues1.

Effectiveness Analysis
Effectiveness analysis entails examining the impact of an intervention on the behavioral, biologic, social, or policy outcomes that the intervention was designed to affect. These analyses are often considered "outcome evaluations." However, they include both effectiveness analysis (impact on these measures in the context of true to life service provision, such as an actual intervention in the field), and efficacy analysis (impact on these measures in an intervention being implemented under ideal or "gold standard" conditions). Efficacy analysis typically includes a direct comparison of intervention approaches made, such as would be seen in a multi-arm research study. Health economists frequently discuss the need to understand the "counterfactual" argument for this sort of analysis. "Counterfactual" is a term used by health economists to describe what the finding is being compared to.

Issues that frequently arise in effectiveness and efficacy analysis are the ability to attribute cause and effect, and the ability to compare the intervention effect to other interventions. As discussed in Chapter 1, "Conceptual Approach and Framework for Monitoring and Evaluation," it is often difficult in effectiveness studies to rigorously attribute the outcomes of interest to the intervention activities unless controls and randomization are included in the study design, which is rarely done in real-life evaluations. If trials occur without controls and randomization of study subjects, it is possible that changes in outcomes identified are due to "secular changes," which are basically unmeasured societal level changes that occur in a group of study subjects. For example, AIDS mortality has an impact on the behaviors of people in a society because they witness the impact of HIV risk, and thus, behavior tends to change over time naturally in a society even without significant interventions. Without a control or comparison group in an evaluation, these changes appear to be intervention effects. Some interventions are especially difficult to evaluate for effectiveness due to the diffuse nature of the intervention, such as with mass media. The target populations are exposed to so many influences and are so dispersed that it is difficult to attribute causation to the intervention. Social and policy interventions can also be difficult to evaluate for similar reasons, as it is difficult to know which of many factors affected a change.

Data needed to conduct an effectiveness analysis include risk behavior, demographic information, intensity, and time of service use. Data can be collected from surveys of individuals who received the intervention and from biologic markers such as sexually transmitted infection (STI) rates. However, it is important that data for such evaluations be collected before and after the intervention. Process data, such as the number of condoms sold, can also be used to infer outcomes of interest, such as the usage rate of condoms. Problems that often arise include having only cross-sectional data on the study population at baseline and follow up, a lack of baseline data, lack of uniformity of measures assessed across intervention types, biased sampling, and collection of only process data. Additionally, the many factors that contribute to quality research (for example, validity of measures, self-report bias, or training of interviewers) are often lacking in evaluation studies. It is also often difficult to assess the rigor of the evaluation data retrospectively because the people who implemented the studies are often not accessible and records on study methodology are not available.

Effectiveness Modeling
Effectiveness modeling involves using mathematical models to estimate the likely level of disease outcomes that will result from the presence of different behavioral and environmental factors. Modeling is valuable for policy purposes because it harmonizes the outcomes of different interventions (an STI intervention may be concerned with the outcome of STI cases treated, while a media intervention may focus on outcomes relating to number of partners), and presents the intervention impact in terms of the disease outcome. Such models require data from behavioral surveys as well as assumptions gleaned from literature on the epidemiology of the disease of interest and the environment. For estimating HIV infections averted, there are some good basic models that can make estimations of the number of people who will become infected with HIV over a set time frame. The AVERT model, available from Family Health International, is one example (Chapter 15, "Translating Survey Data into Program Impact: the AVERT Model," provides more detail on this model)2. Data needed for the models normally include the following:

• HIV prevalence of the target population;
• HIV prevalence of the sexual partners of the target population (usually from population-based surveillance studies);
• average number of sexual partners for the target population (from survey data);
• average number of sexual acts per contact (from survey data);
• condom use rate (from survey data);
• effectiveness of condoms, (from survey data or assumptions); and
• the infectivity of HIV, or the probability of a transmission of HIV occurring in a single sexual contact between an infected and uninfected person; this is affected by STI rates, so data on these are also helpful (from literature and STI surveillance).

The time frame for each parameter needs to be predetermined, but is normally set to one year. By estimating the HIV rates from baseline data and again at follow up, it is possible to take the difference between the two values to then calculate the intervention impact on HIV incidence (for example, the number of HIV infections averted). In controlled studies, the impact of the intervention can be controlled for differences across study arms. More detailed analysis can be conducted with spreadsheets. This model uses probability formulas that convert behavioral measures into estimates of HIV-1 incidence and it has been shown to have a high degree of predictive validity in high HIV-1 prevalence settings3.

Difficulties frequently encountered in modeling exercises include a lack of needed data, the need to annualize behavioral parameters (for example, convert monthly rates of condom use to annual estimates), unknown STI rates, uncertainty in HIV and STI rates among sexual contacts, and lack of consensus on the infectivity of HIV. Studies have shown that when sexual behavior data are collected carefully, there is a high degree of validity in their measurement4,5. To help deal with the uncertainty in many of these parameters, a sensitivity analysis can be conducted using different combinations of the values and examining the impact on the outcome. By running the model with different likely values (extreme values, for example) it is possible to present a range of outcomes that are most likely, and therefore convey the level of uncertainty in the model. One good technique is to use 95 percent confidence intervals from survey data for extreme values in sensitivity analysis. Sensitivity analysis can become very complex, because multivariate interactions are not easy to identify, and they often occur between parameters. Special decision analysis software packages, such as At-Risk, are available for effectiveness modeling, which can be used with spreadsheet analyses6.

Cost-effectiveness Analysis
Cost-effectiveness analysis involves coupling the intervention cost per client with effectiveness measures (both described above)7. Various outcomes can be calculated, depending on the availability of data. For example, recent cost- effectiveness analyses have been conducted of the impact of nevirapine on HIV transmission from mother to child, and on the impact of enhanced STI services on reducing HIV transmission8,9. Below is a summary of the most commonly used outcomes:

• Cost per unit of behavior or change–This includes outcomes such as the cost per condom used or the cost per partner reduced. This approach is not very common in the literature.

• Cost per HIV infection averted–This is a frequently used approach.

• Cost per QALY or DALY saved–Society may place a greater value on averting HIV cases among persons with certain characteristics (especially with regard to the quality of life they will experience) and among persons who will have a longer life should infection be averted. Thus, techniques exist to weight the impact of the infection averted and take into account the quality of life and the number of years of saved life that will result from an intervention. The two most common approaches is to convert the HIV infection averted to Quality Adjusted Life Years (QALY) or Disability Adjusted Life Years (DALY)10. To convert the number of HIV infections averted into QALY or DALY requires that the age of infection of the target population be identified. Discrete stages of infection are then identified, each with an associated time duration and weight for the quality of life. The weighted number of years of life saved from the intervention are calculated using the age specific distribution of the target population. There is a growing consensus in the field on how to do this for HIV. However, most applications have been based on U.S. data and information on the natural history of HIV in developing countries is not well understood due, in particular, to the lack of long-term cohort studies that include quality of life measures.

• Cost utility analysis–This approach uses cost-effectiveness measures described above (mostly QALY and DALY) and takes into account the treatment costs of HIV at different stages. Simple formulas can be used to calculate the cost-utility ratio, which is a common measure used in health economics10. The benefit of the cost-utility ratio is that it places the cost-effectiveness of the intervention in relation to the cost-saving from treatment that result. This allows for direct comparisons to other health interventions.

Assessing the Analysis Strategy
Table 17-1 summarizes the various procedures that are possible for effectiveness analysis. This assessment tool can be used to determine the type of analysis that is possible with different types of data. In some cases, there are options for the outcome variable that is examined. The minimum level of data required is highlighted with asterisks. As described above, four primary types of analysis are possible: (1) cost analysis, (2) effectiveness analysis, (3) modeling of effectiveness, and (4) cost-effectiveness analysis. Cost-effectiveness analysis can be broken out into four additional areas, including: (1) cost per unit of behavior change, (2) cost per HIV infection averted, (3) cost per QALY or DALY saved, and (4) cost-utility analysis, which incorporates the treatment costs averted from the intervention. The next section looks more closely at cost-effectiveness analysis and reviews the steps involved in estimating the cost-effectiveness of an HIV intervention.

Guidelines For Conducting A Cost-Effectiveness Analysis

In this section, the steps normally taken to conduct a cost-effectiveness analysis are reviewed. These follow the recommendations of Haddix and colleagues1, with special attention given to HIV interventions.

Framing the Problem
It is crucially important to frame the problem carefully before initiating a cost-effectiveness analysis. This involves specifying the study question, which in turn helps to define the other key elements of the analysis: perspective, time frame, and analytic horizon. Example of study questions include:

• What is the most cost-effective behavioral intervention for in-school sexually active adolescents in Western Kenya?
• Is it more cost-effective to target older or younger adolescents in an HIV peer education intervention?
• Are peer education programs more cost-effective than HIV voluntary counseling and testing among factory workers in Thailand?

The perspective of the analysis relates to the question, "Who is responsible for the costs and consequences of the program being evaluated?" For example, analysts may be interested in examining the perspective of the donor (such as USAID or the national government) who pays for the intervention. Other perspectives that might be examined include the entire society, the government, the implementing agency, and the individual who receives the intervention. The choice of perspective helps to determine the types of costs that are captured. For example, a donor may incur the costs of the intervention, but not the costs of treatment for HIV infections in a given setting.

The time frame for the analysis should then be determined. In evaluating HIV interventions, a 1-year time frame is typically used. Thus, costs are generated that reflect the annual cost per client, and the annual number of HIV infections averted. It may be useful to also expand the time frame to include more long-term impacts, although it is often difficult to access data on long-term impacts of HIV interventions. In situations in which the time frame is a number of years, it is necessary to discount the costs to the present year value. There are techniques and formulas for discounting, and one simple approach is to use the following formula: (1 + r)-t, where "r" is the discount rate (usually set to 3% to 5%), and "t" is the number of years from the current year.

The analytic horizon should next be determined. This is the amount of time over which the outcome is examined. For example, an HIV intervention may operate for only 2 years, but the benefits of the intervention may be realized over a lifetime for the recipients of the intervention. Thus, the analytic timeframe should probably be 3 years in this instance. For HIV interventions, care should be taken to assure that adequate data are available regarding outcomes over time. In most cases, it is probably better to set the time horizon at a shorter interval to generate a more conservative analysis. If long-term estimates are made, the results may have poor validity, and will be difficult to defend and use for policy purposes.

Identifying the Options to be Compared
Once the study question has been determined, it is important that the comparisons for the analysis be carefully defined. In many cases, an HIV intervention will be compared to no intervention. Alternatives include comparisons between two different interventions, or comparisons between two or more target populations. Comparisons can also be made between the health benefits of an HIV intervention versus a non-HIV health intervention. Various references are available to assist in making such comparisons11. It is important that data for the analysis defined at this step be available.

The more complex the comparisons defined, the more difficult it will be to collect the requisite data for the analysis. Additionally, for purposes of analysis, it is typically necessary to have, at a minimum, pre-intervention and post-intervention data on the effectiveness of the intervention. That means that if two intervention approaches are being compared, four data sets are needed–data for before and after the intervention for each intervention approach.

Identifying the Outcome Measures
This is perhaps one of the most difficult steps in a cost-effectiveness analysis for HIV interventions. Some of the options for outcomes were reviewed earlier in this chapter. Table 17-1 is useful in determining which outcome measure is appropriate and what data are required to use each outcome. Again, some of the most frequently used outcome measures for HIV intervention cost-effectiveness analysis include the cost per HIV infection averted, the cost per quality adjusted life year saved, and the cost-utility ratio. The cost to the client may also be of interest in these analyses. Additionally, it may be useful to examine medical and social costs and benefits of the intervention, although these are often difficult to measure.

Identifying Intervention and Outcome Costs
Next, the costs of the intervention and outcomes need to be estimated. These are reviewed in Chapter 16, "Guidelines for Assessing the Economic and Financial Costs of HIV/AIDS Prevention Programs." Care should be taken to harmonize the costs estimates with the comparisons made, and the analytic horizon of the analysis. With HIV intervention studies, the costs of the intervention are typically conveyed in terms of the cost per client to receive the intervention.

Constructing a Decision Tree
Decision trees are a graphic way to present the data of the cost-effectiveness analysis. They can be quite complex or relatively simple. Figure 17-1 shows a simple decision tree for a hypothetical analysis. In this case, two HIV interventions are being analyzed, Peer Education and HIV Voluntary Counseling and Testing (VCT). The peer education intervention costs $22,000 to reach 1,000 clients, and the VCT intervention costs $29,000. Using actual HIV incidence studies, or the AVERT model, it is found that peer education averted 22 infections (45 minus 23), while the VCT intervention averted 35 HIV infections (45 minus 10) over a 1-year period. Thus, this simple decision tree shows that the cost per HIV infection averted for Peer Education is $1,000 ($22,000 ÷ 22 infections averted). For VCT, the cost per HIV infection averted is $829 ($29,000 ÷ 35).

So while the peer education intervention was less expensive, it was also less effective, and in sum, it turned out to be less cost-effective. However, it is important to realize that interpreting such results also needs to take into account social and policy considerations. It may be that in this setting there are serious social outcomes of learning that one is infected with HIV and thus, there is a desire to avoid HIV testing. Sophisticated decision trees allow for the inclusion of attitudes toward the outcomes, and inclusion of social and policy considerations.

Conducting a Sensitivity Analysis
The final step in the cost-effectiveness analysis is to conduct a sensitivity analysis of the decision tree and its associated analysis models in situations where outcomes are modeled. Sensitivity analysis takes into consideration any uncertainty that occurs in the data used. In all scientific studies there is some level of uncertainty in data that are collected and used for analysis (the speed of an atom, the size of a tumor, the chance of a volcano eruption). One way to capture and analyze the effect of such uncertainty in data is to first model the system mathematically, and to then make systematic changes in the parameters used in the mathematical analysis to see how they affect the outcome. By varying uncertain values over a reasonable range, it is possible to examine changes in variables in the system and see how stable the system is when values are changed. Selecting a reasonable range of values to represent the uncertainty is an important qualitative process that must be conducted carefully.

An example of an important sensitivity analysis that needs to be conducted with each intervention is the likely range of effectiveness that will be generated from the intervention. One good possibility is to use the confidence intervals of the outcome of interest, such as HIV incidence, to set the high and low values for the sensitivity analysis. In situations where one models the outcome, confidence intervals can be used for the input parameters, such as with percent condom use with the AVERT model. The process for conducting sensitivity analysis is to run the analysis multiple times with varying parameter values. This simple form of sensitivity analysis (known as a one-way sensitivity) examines how changes in individual variables affect the outcome of interest. More sophisticated sensitivity analyses can be conducted that examine how sets of input parameters act together to affect the outcome. These typically are conducted with specialized software, such as At-Risk6.

Presenting the Results
Once all of the requisite analysis is completed, it is important to develop a policy presentation of the results. In doing this, it is important to consider the audience for the information. Overly technical presentations to persons not familiar with cost analysis can result in a poor response. It is also helpful to give concrete examples and to present the following details, as recommended by Haddix and colleagues1:

• the study question;
• the study perspective, time frame, and analytic perspective;
• the assumptions used to build the model and estimate outcomes;
• a description of the interventions;
• evidence of the effectiveness of the
• interventions;
• identification of the relevant costs,
• including whether productivity costs are included, and the discount rate used;
• .results of the analysis showing the
• comparisons made;
• results of the sensitivity analysis;
• discussion of the results that incorporate the social and policy perspective; and
• recommendations for action.

Conclusion

Cost-effectiveness analysis can provide important insights into the utility of HIV intervention programs. Care needs to be taken to select an analytic approach that matches the specific research question of interest and available data. When conducting a cost-effectiveness analysis of an HIV intervention, it is important to carefully develop the analysis plan as has been described in this chapter. The approaches outlined here describe basic methods for conducting a cost-effectiveness analysis. More sophisticated techniques not covered in this chapter are also available. It is recommended that use of advanced techniques be done cautiously and with the technical assistance of persons with experience in this area.

References

1. Haddix AC. Prevention effectiveness: a guide to decision analysis and economic evaluation. New York: Oxford University Press; 1996.
2. Rehle T, Saidel T, Hassig S, et al. AVERT: a user friendly model to estimate the impact of HIV/sexually transmitted disease prevention interventions on HIV transmission. AIDS 1998;12(Suppl 2):S27-S35.
3. Weinstein MC, Graham JD, Siegel JE, Fineberg HV. Cost-effectiveness analysis of AIDS prevention programs:
4. concepts, complications, and illustrations. In: Turner CF, Miller HG, Moses LE, editors. Confronting AIDS: sexual behavior and intravenous drug use. Washington (DC): National Academy Press; 1989. p. 471-499.
5. Coates T, Furlonge C, Mwakagile D, et al. Validation of self-reported sexual risk behavior with STD incident rates: results from the voluntary HIV counseling and testing study. From: XIIth International Conference on AIDS. Geneva; 1998. Abstract no.: 14107.
6. Kauth MR, St. Lawrence JS, Kelly JA. Reliability of retrospective assessments of sexual HIV risk behavior:
7. a comparison of biweekly, three-month, and twelve-month self-reports. AIDS Educ Prev 1991;3(3):207-214.
8. At-Risk [program]. 3.5.2 version. Newfield (NY); 1997.
9. Drummond M, Stoddard G, Torrence G. Methods for the economic evaluation of health care programs. New York: Oxford University Press; 1987.
10. Marseille E, Kahn JG, Mmiro F, et al. Cost effectiveness of single-dose nevirapine regimen for mothers and babies to decrease vertical HIV-1 transmission in sub-Saharan Africa. Lancet 1999;354(9181):803-809.
11. Gilson L, Mkanje R, Grosskurth H, et al. Cost-effectiveness of improved treatment services for sexually transmitted diseases in preventing HIV-1 infection in Mwanza Region, Tanzania. Lancet 1997;350(9094):1805-1809.
12. Gold MR, Siegel JE, Russell LB, Weinstein MC, editors. Cost-effectiveness in health and medicine. New York: Oxford University Press; 1996.
13. Murray CLJ, Allan DL, editors. The global burden of disease, Cambridge (MA): Harvard University Press; 1996.