Module 8: Measuring for Impact Overview

 

 

• What is the impact of ECR?
• What is working, and what needs to be improved?
• What adjustments need to be made?

In seeking to reduce the overall rate of HIV infection, programs and interventions for ECR are designed to reach larger numbers of people in expanded geographic areas. Effective and appropriate systems must be in place to monitor programs and evaluate their impact to determine whether the goal is being achieved. Monitoring progress is particularly important because it allows program managers to make adjustments to programs along the way to ensure desired impacts.

Monitoring and evaluation systems are key to ECR. They allow program managers and policymakers to assess whether existing programs and interventions are sufficient and to determine challenges to achieving successful outcomes.

Comprehensive guidelines have been developed by UNAIDS for monitoring, evaluation and surveillance, National AIDS Programmes: A Guide to Monitoring and Evaluation and Guidelines on Second Generation Surveillance. These documents provide detailed information on indicators for prevention and care, as well as methods countries can use in monitoring, evaluation and surveillance.

Module 8, Measuring for Impact:

• Summarizes aspects of the UNAIDS guidelines that are particularly relevant to the ECR process, including monitoring, evaluation and surveillance systems and their different roles.
• Describes appropriate national indicators for tracking change.
• Discusses special issues in ECR monitoring and evaluation.
• Outlines new challenges for monitoring and evaluating care and support programs that are designed to reduce mother-to-child transmission (MTCT).
• Offers a model for estimating the impact of increased coverage.
• Lists key implementation questions for ECR monitoring and evaluation.
• Provides a case study of ECR monitoring and evaluation.
• Suggests resources for further reading.

Monitoring, Evaluation and Surveillance – What Is the Difference?

The terms monitoring, evaluation and surveillance are often used interchangeably, which can create confusion about their meaning. Monitoring and evaluation probably generate the most confusion because they are often used in project management.

• Monitoring is a critical component of a comprehensive program evaluation and is synonymous with process evaluation. Monitoring involves proactively checking to determine whether planned project inputs and outputs are occurring, identifying barriers to achieving the inputs and outputs, and making changes to keep the program on course. It also includes checking the quality of interventions to ensure that best-practice standards are maintained.

• Outcome and impact evaluation, while important, may not be components of all programs because not all programs need to collect data on the effects of a singular intervention. Most often, individuals benefit from and respond to multiple interventions that reinforce each other. It can be more effective to measure the combined effects of multiple interventions using existing HIV, STI and behavioral surveillance systems (see case study). Figure 1 illustrates monitoring and evaluation priorities for ECR.

• Surveillance is most likely outside the domain of individual programs and refers to methods used to track trends in the epidemic and in contributing risk behaviors. UNAID's Guidelines on Second Generation Surveillance recommends implementing a systematic set of HIV, sexually transmitted infection (STI) and behavioral surveillance surveys, which are appropriate to the country's epidemic and repeated on a regular basis. The more limited the epidemic, the more focused the surveillance system will be in high-risk groups, with occasional surveys of the general population to check epidemic spread. The more advanced and widespread the epidemic is, the more extensive the surveillance systems must be to measure trends in the general population.

HIV, STI and behavioral surveillance that provide outcome and impact measures are usually the domain of national ministries of health, which may collaborate with local health authorities. Sentinel HIV and STI sites must be strategically located where HIV transmission is expected to rise because of the high prevalence of risk behaviors in a country. Similarly, behavioral surveillance can consist of selected population subgroups that are sampled regularly and systematically from large geographic areas, such as provinces or major cities.

The responsibility for monitoring program quality and process-level inputs and outputs belongs with the organizations that implement these programs and with the local health authorities who oversee them. At this level, adequate staff must be trained, commodities such as drugs made readily available, communication messages developed for targeted groups, and people with STIs treated appropriately. These factors can be assessed by collecting process data and monitoring service quality with rapid qualitative methods.

National Indicators for Tracking Change
National indicators to track change are complex because of the multi-sectoral response required for HIV prevention and care. They range from the policy environment to direct interventions to blood safety. Table 1 shows a set of indicators developed for national AIDS programs by UNAIDS and collaborating organizations; they fall into one of two epidemic categories:

• Priority generalized epidemics – for countries with greater than 1 percent HIV population prevalence, as measured with proxy samples such as antenatal care clinic attendees.
• Priority concentrated or low-level epidemics – for countries with less than 1 percent HIV population prevalence.

A subset of epidemic category indicators – core indicators and additional indicators – appears in Table 1 for both priority generalized and priority concentrated/low-level epidemics to help countries expand monitoring if resources are available. References, source information for instruments and information on constructing indicators are included at the end of this module.

Countries must select indicators for monitoring prevention and care programs before developing systematic surveys (including quantitative and qualitative studies) for each indicator. The instruments referenced in Table 1 have been tested by UNAIDS and others in international settings and can be adapted for use in different environments.

Countries may decide to construct regional/area or national sampling frames to measure indicators and develop estimates. A regional approach can be useful in large countries such as China or India or where factors suggest that obtaining estimates in several areas is more helpful than one national estimate. A national sampling can be powerful because it yields easy-to-understand estimates and attracts the attention of important policymakers and international donors. But national indicators, because they reflect the performance of many interventions in one large figure, sometimes produce information that is less useful than regional indicators. Regional estimates give the data greater relevance at the local level, where the information may have more influence and produce a response. Regional indicators, and those of even smaller areas, require more extensive data collection because samples must be large enough in each region to track changes.

These instruments can be found at the following Web sites:

• WHO: www.who.org
• UNAIDS: www.unaids.org
• Family Health International: www.fhi.org
• UNICEF: www.unicef.org
• Macro International (DHS): www.macroint.com
• University of North Carolina (MEASURE): www.cpc.unc.edu/measure/

Table 1: Program Areas with Core and Additional Indicators
Program Area Indicator/Policy	Toots for Measurement	C=Core Indicator	A=Additional Indicator
		Priority Generalized Epidemic	Priority Concentrated Low Level
Policy
AIDS Programme Effort Index (API)	API questionnaire and protocols	C	C
Spending on HIV prevention	Under development
Condom Availability and Quality
Condoms available nationwide	WHO condom protocol (P12)	C	C
Condoms available retail	MEASURE Evaluation*/WHO/PSI Compiled Condom Availability and Quality Protocol	C	A
Condom quality	MEASURE Evaluation/WHO/PSI Compiled Condom Availability and Quality Protocol	C	C
Stigma and Discrimination
Accepting attitudes toward HIV+ people	Surveys (UNAIDS, DHS, FHI, UNICEF)	C	C
Employers not discriminating	UNAIDS protocol on discrimination	C	C
Knowledge
Knowledge of HIV prevention	Surveys (UNAIDS, DHS, FHI, UNICEF)	C	C
No incorrect beliefs about HIV	Surveys (UNAIDS, DHS, FHI, UNICEF)	C	C
Knowledge of HIV prevention among MSM	FHI BSS among MSM		C
Knowledge of HIV prevention among IDUs	FHI BSS among IDUs		C
Knowledge that MTCT can be prevented	Surveys (UNAIDS, DHS, FHI, UNICEF)	C
Voluntary Counseling and Testing (VCT)
People who requested test and received results	Surveys (UNAIDS, DHS, FHI, UNICEF)	C	A
Districts with VCT services	District assessment — no specific tool	C
Quality of counseling and referral	UNAIDS VCT protocol	C	A
VCT centers with minimum conditions	UNAIDS VCT protocol	C	A
Quality of VCT laboratories	WHO testing protocol, blood safety protocol Surveys (UNAIDS)	A	A
Mother-to-Child Transmission (MTCT)
Pregnant women counseled and tested	UNAIDS MTCT protocol; VCT protocol	C
ANC clinics offering and referring for ANC	UNAIDS MTCT protocol; VCT protocol	C
Quality HIV counseling for pregnant women	UNAIDS MTCT protocol	A
Provision of ARV therapy	Surveys (UNAIDS, DHS, FHI)	A
Sexual Negotiation and Attitudes
Women's ability to negotiate safe sex	Surveys (UNAIDS, DHS, FHI)	A	C
Sexual Behavior
Higher-risk sex in the last year	Surveys (UNAIDS, DHS, FHI)	C	C
Condom use at last higher-risk sex	Surveys (UNAIDS, DHS, FHI)	C	C
Commercial sex in last year	Surveys (UNAIDS, DHS, FHI)	A	C
Condom use by clients at last paid sex	FHI BSS for sex workers	A	C
Condom use by sex workers with last client	FHI BSS for sex workers	A	C
Higher risk male-male sex in last year	FHI BSS for men who have sex with men		C
Condom use at last anal sex between men	FHI BSS for men who have sex with men		C
Young People's Sexual Behavior
Median age at first sex	Surveys (UNAIDS, DHS, FHI BSS-youth)	C	A
Young people having premarital sex	Surveys (UNAIDS, DHS, FHI BSS-youth)	C	A
Condom use at last premarital sex	Surveys (UNAIDS, DHS, FHI BSS-youth)	C	A
Young people with multiple partners	Surveys (UNAIDS, DHS, FHI BSS-youth)	C	A
Condom use at last higher-risk sex	Surveys (UNAIDS, DHS, FHI BSS-youth)	C	A
Condom use at first sex	Surveys (UNAIDS, DHS, FHI BSS-youth)	A	A
Age-mixing in sexual relationships	Surveys (UNAIDS, DHS, FHI BSS-youth)	A	C
Injection drug use
Injection drug users sharing equipment	FHI BSS for injection drug users		C
Injection drug users never sharing equipment	FHI BSS for injection drug users	C
Drug injectors using condom at last sex	FHI BSS for injection drug users		A
Blood Safety / Nosocomial Transmission
Screening of blood units for transfusion	MEASURE blood safety protocol	C	C
Reduction of blood transfusions	MEASURE blood safety protocol	C	A
Districts/regions with blood bank	MEASURE blood safety protocol	C	C
Accidental transmission in health care settings	MEASURE service provision assessment (SPA)	A
STI Care and Prevention
Appropriate diagnosis and treatment of STI	WHO/UNAIDS STI facility survey	C	C
Advice on prevention and HIV testing	WHO/UNAIDS STI facility survey	C	C
Drug supply at STI care services	WHO/UNAIDS STI facility survey, SPA	C	A
Treatment seeking for STI	Surveys (UNAIDS, DHS, FHI)	A	C
Care and Support
Medical personnel trained in AIDS	MEASURE SPA, training statistics	A	A
Health facilities with capacity to deliver care	WHO protocol for care and support	C
Health facilities with drugs in stock	WHO protocol for care and support, SPA	A
Households helped with care of young adults	Survey (UNAIDS)	C
Households helped with care of orphans	Survey (UNAIDS)	A
Health and Social Impact
HIV prevalence among pregnant women	WHO/UNAIDS protocols for surveillance	C	C
Syphilis prevalence among pregnant women	WHO/UNAIDS protocols for surveillance	C	C
HIV prevalence in sub-populations at risk	FHI sampling manual	A	C
Prevalence of orphanhood	Surveys (UNAIDS, DHS, UNICEF)	C
Schooling of orphans	Surveys (UNAIDS, DHS, UNICEF)	A

*Monitoring and Evaluation to Assess and Use Results (MEASURE) is a USAID-funded program at the University of North Carolina at Chapel Hill.

Source: National AIDS Programmes (2000).

Special Issues in Monitoring and Evaluating ECR

Changing Populations and Changing Prevention and Care Needs
It is important to understand that populations for which HIV programs are designed are fluid, not static. A population's HIV prevention needs evolve constantly because of changes in the population, such as geographic movement and behavioral practices. What works at one time may not work in the future. Some examples of changing populations and their changing prevention and care needs are:

• Commercial sex workers (CSWs) and injection drug users (IDUs) move frequently to avoid contact with police or other authorities. Commercial sex areas in certain cities are raided (sometimes because of attention focused on them by the HIV epidemic and prevention programs), forcing CSWs to move to street corners, private homes and parks. Similar movements can occur among IDUs. Prevention interventions must follow the movement of target population groups.

• As the epidemic changes, population groups require different prevention messages and programs. Prevention interventions for men who have sex with men (MSM) in many Western countries were considered successful in the late 1980s after substantial behavioral change and reduced HIV infection rates were documented. Shortly after this success, however, research began to show MSM relapsing into unsafe sexual practices and new cohorts of younger men, many of whom had not experienced the epidemic directly, engaging in unsafe sex. These changes necessitated a new wave of targeted prevention messages and programs.

• Migrating populations present unique challenges to prevention programs because they move so frequently. The most effective places to reach migrating communities (at source areas, along the migration route or at a destination) must be determined.

An effective way to address populations' changing prevention and care needs is to conduct "mapping" (identifying target groups' locations and learning how and where they move) and develop an ongoing system of in-depth interviews with selected target groups. It is important to incorporate a system of direct annual contact with target group members that includes mapping and in-depth interviews, particularly where interventions are being expanded or with transient target groups, such as migrating populations or IDUs. These approaches help programs stay abreast of the movements of target groups, their size and needs, and to assess risky behaviors and behavior change. These methods also help programs determine where target populations move (so that interventions can move with them); what challenges must be addressed to achieve success; and whether individuals are responding to prevention interventions.

Capacities of Organizations
As programs expand, they require trained staff and more commodities, such as condoms, STI drugs and laboratory equipment. Not all countries have enough trained staff or access to the required commodities and equipment to provide the level of prevention and care services needed.

It is important to address capacity and commodity shortages so that prevention and care services can continue uninterrupted to targeted population groups. All countries can undertake monitoring of organizational capacity and available commodities. Formal assessments of organizational capacity and performance can be conducted to ensure an effective ECR (see Module 5, Human Capacity Development).

In Nepal, for example, a survey of pharmacists was conducted using the simulated client method to determine whether they were using STI syndromic training materials in their drug-dispensing practices. Pre- and post-intervention surveys found that the positive results of the training were only short-term: Pharmacists ultimately reverted to their old practices. This evaluation led to a reassessment of the intervention and revisions in the training materials.

Intervention Coverage
Behavioral research indicates that individuals need a critical mass of interventions that reinforce each other – working toward the same outcome at the same time – for behavior change to be successful. As ECR expands interventions to more population groups over larger geographic areas, it becomes even more critical to determine whether these interventions facilitate behavior change.

An effective way to determine intervention coverage is to implement behavioral surveillance surveys (BSS), which provide reliable measures of HIV risk behaviors in large geographic areas over time. Typically, BSS questionnaires are structured so that the data can help program managers determine what types of behaviors are changing and what types are not. For example, men may begin to use condoms with CSWs but not with their non-paid casual partners – a specific risk factor that prevention interventions should address. If behaviors are changing, then BSS can help document successful interventions.

BSS also can be used to assess the coverage of interventions by incorporating questions on exposure. Persons may be asked about visiting an STI clinic, receiving a condom from an outreach worker, or recalling a particular communication message.

If BSS samples do not cover geographic areas where measures of intervention coverage are desired, then special "rapid coverage surveys" can be implemented. These surveys follow methodological guidelines similar to other surveys attempting to construct a representative sample, but they focus on exposure to HIV prevention and care programs. Low-measured exposure to programs or messages that have been targeting wide audiences for several years might suggest that coverage is inadequate and that program managers should reassess intervention content and reach.

Challenges in Monitoring and Evaluating Care and Support and Prevention of Mother-to-Child Transmission (MTCT) Programs
The increasing number of PLHAs, particularly in Africa, has demanded that care programs be expanded to serve them and their families, including the millions of children who are orphaned. Reductions in drug prices have enabled care programs in some countries to incorporate medication dispensing where the facilities and staff to administer drugs and conduct monitoring exist. Drugs to reduce MTCT are used in many developing countries, although coverage is far from universal.

The capacity to deliver care is an issue in monitoring and evaluation. Health delivery systems in many developing countries were stretched thin even before the HIV epidemic. Countries can look for assistance from existing community-based systems of care; however, the capacity of these systems to provide care may be limited as well.

Because countries select indicators based on their care and support goals, indicators may vary considerably from country to country. The challenge for countries is to adapt instruments that are feasible to implement. For example, a representative survey of HIV-infected persons can provide many indicators on care and support, but may be difficult to conduct because of confidentiality or other concerns. Likewise, knowing the proportion of HIV-infected mothers not breastfeeding might help determine if a prevention program for vertical transmission is successful, but gathering such data may not be possible.

An alternative method for obtaining indicator information is to estimate the number of HIV-infected persons (or mothers, depending on the service) in a given area, using modeling programs based on surveillance or ad hoc data. This establishes a denominator from which to measure achievements in service provision. For example, if 1,000 PLHAs are estimated to live in a certain region and all programs report serving 500 persons combined in that region over a given period, an approximate indicator of reach to PLHAs in that region would be 50 percent.

Another challenge is assessing quality of care because no gold – or minimum – standard currently exists for most countries. Standards for services must be developed to monitor implementation effectively and establish internationally comparable indicators.

Research on how best to monitor and evaluate care and support interventions is needed over the next few years. For now, countries must set realistic monitoring and evaluation goals that reflect available resources and capacity. These goals can be adjusted as research produces "best practices" for monitoring and evaluating care and support programs and as funding and capacity increase.

Estimating the Impact of Increased Coverage: HIVTools
A user-friendly modeling tool developed for UNAIDS by researchers at the London School of Hygiene and Tropical Medicine helps programs understand the impact of increased coverage on HIV incidence and prevalence. HIVTools uses epidemiological, behavioral and coverage parameters to estimate the interventions' impact on specific target groups. Four models are available for interventions aimed at CSWs, IDUs, school-age and adolescent youth, and blood transfusion services. Additional models are under development.

The model SexWork, for example, simulates the impact of HIV transmission interventions that target CSWs and their clients. It requires the user to supply information, including:

• size of the overall sex worker population
• proportion of sex workers targeted by the intervention
• proportion of those targeted that has been reached recently by the intervention
• data on condom use among sex workers who have been reached recently by the intervention
• data on condom use among sex workers who have not been reached recently by the intervention

Once the requested data are in place (default values also can be used), HIVTools' software generates data and graphs on probable future trends in HIV prevalence, incidence, and cases averted among sex workers and their clients.

Figure 2 shows how HIV prevalence is affected by increasing the coverage of a sex worker intervention from 25 percent of a population of 5,000 sex workers to 80 percent, when the rate of consistent condom use in the reached population grows from 25 percent to 90 percent. In the lower-coverage intervention, the model predicts that 1,533 cases of HIV among sex workers will be prevented and that prevalence over a five-year period will rise from 20 percent (the baseline level entered into the model) to 53 percent. In this example, the limited coverage does not stop HIV from infecting more than half of the population.

If effective coverage is increased to 80 percent of the sex work population with the same behavioral changes, the estimated number of averted HIV cases rises to 4,698 and prevalence decreases from 20 percent (again, the baseline percentage used in the model) to 10 percent. This scenario illustrates the importance of coverage in preventing an epidemic.

Using Data to Effect Change in Policies and Programs
Surveillance, evaluation and monitoring data are most effective when used to develop positive changes in policies and in the quality of interventions and programs – and to increase availability of program funds. Data must be distributed routinely to individuals and institutions responsible for policy, planning, resource allocation and budget development. This includes distribution of reports to policymakers, journalists, HIV program managers and other stakeholders. Each person views HIV through a different lens, so data summaries should reflect these differences and construct appropriate, but different, messages. For example, a surveillance report prepared for policymakers would be more concise and contain less scientific jargon than one written for epidemiologists.

Key Implementation Questions for ECR Monitoring and Evaluation

Defining Monitoring, Evaluation and Surveillance

What is the difference between monitoring, evaluation and surveillance? Which methodology(ies) is most appropriate for programs and projects? Which methodology(ies) is most appropriate for national programs?

Indicators for National Programs

Which indicators are used in the country's national program? Based on a country's strategic planning priorities, should any new indicators be collected? How does prioritizing resources for further monitoring and evaluation compare to using resources for program implementation?

Special Issues in ECR Monitoring and Evaluation

I Has the location of target populations been tracked by the program over time? What strategies have been used to track populations? Have any patterns been found in the movement of these populations? Does the program have sufficient capacity to implement monitoring and evaluation? What strategies are being used to strengthen this capacity? Are BSS or rapid coverage surveys being implemented? If not, what type of information must be collected? Is there a monitoring and evaluation approach for care and support programs? For treatment programs? For orphans and vulnerable children (OVC) programs? If not, what information is needed to consider implementing, monitoring, and evaluation for these programs?

Key Questions for Implementing Monitoring and Evaluation for ECR

Key questions for planners and program managers to consider when developing and implementing monitoring and evaluation systems for ECR are featured above.

Further Reading

1. Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries: A Handbook for Program Managers and Decision Makers. 2001. Arlington, Va: Family Health International.
2. Guidelines on Second Generation Surveillance. 2000. Geneva: UNAIDS.
3. National AIDS Programmes: A Guide to Monitoring and Evaluation. June 2000. Geneva: UNAIDS.

Case Study

Case Study: Monitoring and Evaluation of ECR Implementation in Nepal
Nepal is trying to scale-up its interventions to cover key target groups with high-risk behaviors in a larger portion of the country than it works in now. Although Nepal was long thought to be a low-prevalence country, recent data show the country has a concentrated epidemic. HIV prevalence among sex workers nationwide is estimated at 40 percent and, in Kathmandu, 50 percent. HIV among sex workers in Kathmandu increased from less than 1 percent of the population in 1992 to 17 percent in 1999. While sentinel surveillance data suggest that HIV remains low among STI clinic attendees, evidence in the far western part of the country suggests that large numbers of migrant workers returning from high-prevalence areas in India may be bringing HIV back with them, infecting their wives/sexual partners.

In 2000, a consortium of international donors pledged a multimillion-dollar package to reduce HIV transmission in Nepal, prompting the design of enhanced surveillance, evaluation and monitoring systems. Figure 3 shows the current systems in Nepal and a potential design for expanding them. Discussions about enhancing these systems are continuing.

Figure 3 illustrates that projects run by organizations with their own monitoring systems are located in the geographic areas covered by HIV and behavioral surveillance to monitor HIV and behavioral trends. (The map cannot show all the systems that will be in place.)