Evaluating Programs for HIV/AIDS Prevention and Care in Developing Countries

Chapter 3
II. Operational Approaches for Evaluating Intervention Strategies

Barbara Franklin, Donna Flanagan, and Hally Mahler

This chapter will look at the essential evaluation activities performed at different stages of implementing behavior change communication (BCC) projects and discuss critical issues and inherent challenges. Behavior change communication is one component of behavior change interventions (BCI), a broader designation that includes many other types of interventions, such as services, commodity distribution, and community mobilization, in addition to communication interventions.

The evaluation of a BCC project is an integral part of its design and must be considered at different stages throughout the project's development: during the stages of formative research (situation analysis and audience analysis), during pretesting, while monitoring implementation, as well as during the evaluation of communication effects. In other words, evaluation of BCC projects must be considered an integral part of the audience-centered communication cycle, with implications for each of these stages.

Audience-Centered Communication

Audience-centered communication is an approach to BCC based on a model of dialogue between those who wish to promote behavior change and a participating audience. It is essentially a consultative partnership, focusing on interaction at every stage between the communicating institution and the public. While a number of methodological tools are available (focus groups, key informant interviews, and various participatory learning and action [PLA] techniques, among others), no technique will be truly effective unless it is imbued with an attitude of respect for the target audience and a determination to understand the audience's point of view and address the audience's central concerns throughout the BCC campaign, culminating in its evaluation.

Audience-centered communication has three stages. The first stage determines the parameters for effective BCC through a series of research and planning activities. During this stage, communicators look at research already done, and, if necessary, carry out two types of formative research: a situation analysis and an audience analysis. The second stage involves developing a communication concept and messages based on the formative research findings, drafting materials, and pretesting them with the target audience. The third stage involves implementing BCC activities, and includes monitoring communication activities, checking comprehension, and assessing communication effects. While evaluation is often considered a separate phase that follows implementation, experience shows that it is more useful to think of it as an integral part of a communication intervention. This emphasizes the need to generate immediate feedback, verify comprehension, and develop an awareness of effects. The steps of the audience-centered communication cycle are listed below. Activities with a particular relevance to evaluation are in bold face.

I Planning BCC

Review baseline data
Conduct situation analysis
Segment the audience
Designate target audience(s)
Establish behavior change goal
Conduct audience analysis
Develop BCC objectives
Make action plan

II Developing BCC

Develop BCC concept
Develop messages
Choose channels
Develop BCC materials
Pretest BCC materials
Produce BCC materials

III Implementing BCC

Carry out BCC activities
Conduct process evaluation
Assess communication effects

The evaluation activities conducted at each of these stages are described in greater detail below.

Planning BCC

Two types of formative research should be conducted at the planning stage of a BCC project to gather information to guide the project strategies. Planners need to investigate both the individual and community environment before formulating a strategy for behavior change communication. This research is the basis for developing communication objectives–the criteria on which the BCC project ultimately will be evaluated.

Conduct Situation Analysis
The situation analysis (also called community investigation) reveals the parameters within which BCC will take place. It includes an analysis of the:

·         Epidemiological environment–The incidence and prevalence of sexually transmitted infections (STIs), including HIV, among various target audiences.

·         Economic environment–The factors that affect people's purchasing power and spending patterns, which will permit a better understanding of how, for example, prostitution, condoms, and health care are purchased. An understanding of the macroeconomics of a country helps explain the role that the infrastructure, the media, and the market will play in distributing health care messages and facilitating programs.

·         Political environment–The laws, government policies, agencies, and advocacy groups that influence and limit various organizations and activities. The political environment can affect condom advertising, public health, access to health care, the economic power of women, the sex trade, sex education, the operation of STI clinics, and the illicit drug trade.

·         Cultural environment–The institutions and other forces that shape society's basic values, perceptions, preferences, and behaviors. Religion, language, educational institutions, literature, popular music, the press, and theater all play a role in determining the status of women and in influencing sexual behavior, attitudes toward AIDS, and family values.

·         Organizational/development environment–The programs, projects, and interventions that are already in place so that complementary and/or shared efforts can be encouraged and duplication avoided.

The situation analysis tells how the community is organized and how it works–essential parameters within which any BCC activity must function. A carefully conducted situation analysis can save time and money by helping to ensure that BCC activities are chosen well and that they correspond to the realities of the situation, avoid traps, and make best use of their opportunities. It also takes into consideration the need to create a supportive environment, which may lead to a focus on advocacy or community development issues needed to support the individual changes BCC is promoting.

On the basis of the situation analysis, the BCC planning team can segment the audience, and choose target audiences from among the segments for specific BCC activities. A target audience is defined as the group that needs to change its knowledge, attitudes, or behaviors to achieve the program goal. Target audiences for effective BCC include not only groups at risk because of their behavior, but also others who have an impact on the risk situation, including authorities (such as police), gatekeepers (such as brothel owners), and service providers (such as STI clinic staff).

Conduct Audience Analysis
The situation analysis describes the community and larger environment within which change takes place, but ultimately much of the behavior change with respect to HIV/AIDS and STIs takes place on an individual level. Behavior change is usually not a one-time, definitive event, but rather is preceded by a series of steps, including changes in knowledge, shifts in attitude, and tentative preliminary steps to behavior change, such as trying a behavior once. For this reason, developing an effective BCC program requires an in-depth "insider" understanding of the members of each of the target audience segments and where the segment as a whole is on the steps of change at any specific time–their level of knowledge, their relevant attitudes (beliefs, values, preferences, expectations), the barriers to change they face and express, and the factors that might motivate them toward behavior change (as well as their communication preferences, essential information for a strategic choice of channels.)

The audience analysis should seek to provide a clear picture of the interior reality of members of each segment and should provide answers to the following questions:

• How do members of the target audience segments conceive of HIV/AIDS and STIs?
• Where are they in the process of behavior change?
• What is their level of knowledge of the specific facts of HIV/AIDS and STIs, their transmission and prevention?
• What is their estimation of their personal risk of becoming infected with HIV and STIs? If they feel they are not at risk, why not?
• What are their perceptions of peer and social norms governing sexual behaviors and HIV prevention?
• What peer groups and significant others are most important to them?
• What are the behaviors that put them at risk for HIV and STIs?
• How do they currently understand and practice preventive behaviors?
• What do they see as the benefits of changing their behaviors?
• What do they see as the disadvantages of changing behavior? What pressures make it difficult for them to change their behaviors?
• What power do they believe they have to change their behavior, and if it is limited, why, and by whom or what?
• What would it take for them to change to a safer behavior or continue with safe behavior?
• When and where do they usually get information about sexual and health topics?
• How do they communicate with others? Where do they meet? When? Who is in their immediate social network?
• What sources of information on sex and health do they find most credible?
• When and where would be the best times to talk with them about HIV prevention, distribute condoms, make STI treatment services available, and who would do this most effectively?

Develop BCC Objectives
On the basis of the situation analysis and in-depth audience analysis, the BCC planning team then develops communication objectives. These objectives should specify the nature of the change the BCC campaign will engender. They should not be limited to behavior changes but should also include changes in knowledge, attitudes or the decision or intention to try behaviors that precede definitive behavior change. Experience shows, for example, that it is often unrealistic to expect that a BCC campaign will bring about an immediate increase in condom use, but a lack of change in condom use does not necessarily mean that nothing has changed. Important changes may have occurred and may be preparing the way for later behavior change. The process of behavior change is complex and incremental, and BCC planners can best approach it on those terms–like chipping away at a big stone, rather than trying to move it with a single push. Well-formulated BCC objectives recognize these steps of change.

Knowledge changes are the easiest to identify, and as a result, many BCC campaigns focus on increasing knowledge long past the time when lack of knowledge is really the problem for a target audience. Attitude changes are often even more important but neglected by communicators. For example, audience members may change their idea of condoms from a pregnancy prevention device, to something that can keep them safe from STIs and HIV, or they may increase their personal perceived risk of HIV/AIDS. Norms may also change; it may become more usual for women to negotiate the terms of sexual behavior as the result of a BCC campaign, or people may develop new concepts of masculinity, or a sense that condoms can be sexy. There may also be changes in law or policy, or changes in the public discourse, as reflected in the media.

Communication objectives should be stated in the future tense, in terms of changes in knowledge, attitudes, and skills or in the policy environment that will be evident after the BCC activity, and they should directly reflect needs identified through the in-depth audience analysis. Here, for example, are some communication objectives aiming at changes that precede behavior change. Note that they are stated in behavioral terms so that their success is easy to measure:

·         Change in knowledge–After the mass media campaign, audience members will say they cannot get HIV through haircuts, manicures, or mosquito bites, or by sharing clothes or dishes.

·         Change in attitude (beliefs)–After the campaign, when asked what they think of when they hear the word "condom," audience members will more often use words reflecting a positive image of strength and protection.

·         Change in attitude (perceived risk)–After the workshop, audience members will say it is quite possible that they or their friends could get HIV if they have sex outside of a monogamous relationship and do not use condoms.

·         Change in skills–After seeing the video and practicing with the peer educator, audience members will be able to show how they would negotiate condom use with a partner.

·         Change in the discourse–After the campaign, a survey of the popular press will reveal less judgmental language applied to sex workers.

Developing BCC

Pretest Communication Materials
Once the BCC planning team has written communication objectives, developed concepts, chosen the media channel(s), designed messages (that is, the specific words and images) to express the concept, and developed prototype materials to match the objectives, the materials must be pretested with the target audience. Pretesting is an evaluation activity that gives immediate feedback on the effectiveness of BCC messages and materials. It is an early evaluation of the BCC planning process that can and should lead to immediate adjustments in the materials or methods to be used in the BCC project.

Thorough and systematic pretesting is an essential part of marketing and plays no small role in the success of commercial advertising–a success social marketers have emulated. For example, in Nepal's AIDSCAP II project, implemented by FHI, alternative images of a condom were thoroughly pretested to find one that would appeal to a wide spectrum of audience members and would carry a strong association of condoms with AIDS prevention. A number of options were rejected based on the pretests, and the winning image was further refined. The resulting image of "Dhaaley Dai" (Big Brother Condom) shows a smiling, muscular condom character kicking out a small figure representing the virus. It is accompanied by a slogan (rhyming in Nepali): "Wear a condom and drive away AIDS." This message has been widely disseminated in Nepal and has achieved a very high level of recognition.

There are a number of good pretesting techniques, including focus group pretesting, and pretesting through individual or group interviews. Alternative versions of materials should always be pretested, so that the pretest has meaning. While the pretest may look like an informal discussion, it should be structured around a definite set of questions that reveal the different qualities for which the materials are being tested. For example, interviewers could use the question guide in Box 3-1 to pretest a pamphlet with groups or individuals.

Implementing BCC

Conduct Process Evaluation
Process evaluation is conducted throughout the implementation of BCC activities and actively involves key stakeholders, such as project managers, beneficiaries, organization staff, and donors. This participatory approach to process evaluation allows the stakeholders themselves to identify the essential indicators they want to measure and report on and helps to insure that the evaluation will be relevant and useful for designing future activities. Its purpose is to determine whether activities are proceeding according to the plan and if not, to indicate where changes need to be made. Questions asked during process evaluation obviously reflect the activities of the program. They might include such questions as the following (for a peer education activity):

• Were peer educators selected, trained, and supervised?
• Are the peer educators performing the duties that were expected of them?
• Is supervision being conducted as planned?
• Are the communication channels being used as planned?
• Have the radio or TV messages been broadcast?
• Was the target audience involved with message development?
• How many target group members have been reached?

Process evaluation can also examine strengths and weaknesses of an ongoing intervention. What follows is an instrument for assessing and monitoring behavior change communication interventions that sets out standards for effective projects. It then asks specific questions that lead the user to recognize whether the intervention is adhering to the established standards for high-quality BCC interventions. These questions are meant to be asked throughout the implementation of BCC interventions.

Standard 1: Interventions should focus on well-characterized, specific target audiences.

1. Who is the primary target audience for this BCC intervention?
2. Has this primary target group been appropriately divided by segmenting variables? If not, which variables have not been considered that now appear important to the segmentation?
3. Are there other people who influence the primary target group who are not yet being addressed? If yes, who are they?
4. How can the project address these other people?
5. What is the risk behavior(s) that the primary target audience is practicing? What is the desired behavior?

Standard 2: HIV/AIDS prevention interventions and messages must be crafted to motivate and appeal to the specific target audience's perceived needs, beliefs, concerns, attitudes, present practices, and readiness to change.

1. What additional knowledge is needed, what attitudes need to change, and which skills need to be mastered before the target audience will be able to adopt the desired behavior?
2. What are the main messages used in this intervention? Do the main messages address the needed knowledge, attitudes, and skills? If not, what is missing, or what does not match these needs? (To answer this question more specifically, the following additional questions are useful.)
3. What gaps in knowledge appear to influence audience members behavior with respect to HIV/AIDS/STIs? Do the messages match the audience's gaps in knowledge?
4. What does the audience perceive as its most important needs? Do the messages match the audience's perceived needs?
5. What are the audience's main beliefs related to sexuality and HIV/AIDS/STIs? Do the messages make appropriate use of (appeal or respond to) these beliefs?
6. What are the main concerns of the audience members? Do the messages refer to these concerns?
7. What attitudes presently inhibit change in the audience? Do the messages respond to the attitudes and encourage/model different attitudes?
8. What undesirable behaviors do audience members currently have? Do the messages specifically address the disadvantages of these undesirable behaviors?

Standard 3: At-risk individuals must be provided with both skills and supplies to prevent HIV.

1. Are any new skills needed for the audience members to change? If so, what skills?
2. Do the messages model the needed skills?
3. Are supplies that are needed for safe behavior available to all the audience members?
4. Are supplies that are needed for safe behavior affordable for all the audience members?
5. Is STI treatment easily available and affordable for all the audience members?

Standard 4: A supportive environment needs to be created for HIV prevention and for the protection of those infected with HIV.

1. What are the social, cultural, environmental, political, and organizational conditions that may influence the target audience's HIV/AIDS risk behaviors?
2. Does this intervention try to influence these social, cultural, environmental, political and/or organizational factors? For example, does it:
   • support traditional and cultural values that encourage low-risk behaviors?
   • persuade government officials to change public health policies?
   • influence organization/corporate officials to discontinue discriminatory practices or policies?
   • mobilize support among the general public to work for changes in public policy?
   • promote the social acceptability of alternatives to risk behaviors?
   • protect human rights of all people affected by HIV/AIDS?
   • actively fight discrimination?
   • educate the whole community for care, compassion, and prevention?

Standard 5: Mechanisms need to be created to maintain and sustain HIV prevention behaviors and activities over time.

1. Does this BCC intervention include follow-up mechanisms to reinforce and encourage the maintenance of newly acquired attitudes and behaviors? For example:
   • periodic follow-ups and re-certification of peer educators;
   • HIV prevention messages mainstreamed into school curriculum at all grade levels;
   • campaigns to reinforce messages focused on maintaining new behaviors;
   • annual meetings for organizations working in the HIV prevention area;
   • meetings organized to discuss "lessons learned."

Standard 6: BCC planners should identify and use opportunities to work collaboratively and in different sectors of the community/country.

1. Does this intervention actively collaborate with other partners and implementing agencies?
2. Does this intervention take into consideration activities and materials aimed at this target audience by other organizations?
3. Is this intervention designed to involve the resources and expertise of other individuals and organizations in the public and private sectors?

Standard 7: A monitoring plan is essential to guide the adequate implementation of behavior change communication projects.

1. Does this intervention have a monitoring budget?
2. Does this intervention have staff available for monitoring and supervision?
3. Have new directions been identified as a result of monitoring? If so, what are they?

Assess Communication Effects
Monitoring an ongoing BCC program according to the above standards will go far toward developing an understanding of the effectiveness of that program. However, it is also useful to pause, after the conclusion of a BCC program or campaign, to take a close look at the net effect of that program on the target audience. This can be done using quantitative or qualitative methods.

It is possible to conduct a systematic, quantitative evaluation of any change in knowledge, attitude, intention, skill, or reported behaviors (actually a measure of norms) if these are defined specifically enough in the objectives, and if an appropriate methodology is chosen (Section III of this book provides more information on the methodologies for measuring behavioral outcomes).

However, this type of outcome evaluation is often not necessary at the individual project level (see also the section on "Differentiating Evaluation Efforts" in Chapter 1). A simpler method, and one that usually yields richer insights, is to repeat some of the in-depth formative research with different members of the same target audience segments, to identify any evidence of change. This method, called assessment of communication effects, does not attempt to prove that a certain communication campaign caused a certain change (such causality is misplaced, in any case, when dealing with such a profoundly social realm as communication). Rather, it is an attempt to take a second look at the audience, to see how things have changed since the BCC program began.

For example, suppose that a negative image of condoms has been identified as an attitudinal barrier as a result of focus groups or in-depth interviews in which participants referred to condoms in a variety of negative ways, and a campaign has been conducted to change the image of condoms. After the campaign, follow-up focus groups or interviews with the same segments may reveal some change in the discourse on condoms. A striking example from Vietnam came when follow-up focus groups were held after a BCC campaign in which one of the objectives was to give condoms a more friendly, protective, and less medical image. When the groups were asked, "What do you first think of when you think of condoms?" (a question also asked in the original research) many participants responded, "condoms are like a loyal bodyguard." This phrase, echoing a message from the BCC campaign, provided evidence that the campaign had had its intended effect.

Assessing communication effects can also produce data about exposure to media messages, about recall of specific messages, and about the relative impact of different communication channels. An assessment of communication effects not only reflects the success or failure of a previous BCC campaign, but also provides feedback on audience concerns that can be addressed in the next campaign. In this way, the BCC process can become more like a dialogue between the BCC planning team and the audience. Thus, the assessment of communication effects guides future BCC campaigns, closing the planning-implementation-evaluation circle.

Conclusion

This chapter has shown how evaluation is an integral part of the design of audience-centered communication projects, not only at the final stage but also at other key points during the project cycle. This ongoing concern with evaluation from the outset helps to ensure that the communication activities are structured around a dialogue between the members of the target audience and the BCC planning team. Elements of evaluation are included in the situation analysis and audience analysis (formative research), in setting BCC objectives, in pretesting of materials, as well as in the process evaluation (monitoring of implementation) and the final assessment of communication effects. Assessing communication effects allows the audience to give feedback on the campaign as well as on changes within the audience itself relating to the campaign objectives. This then allows the BCC team to plan the next campaign. Thus, the final assessment closes the audience-centered communication cycle and assures that the process maintains its essential character of a dialogue between BCC planners and the audience.

Recommended Reading

1. Bertrand JT. Pretesting communication materials. Chicago: Communication Laboratory, Community Family Study Center, University of Chicago; 1978.
2. Franklin B. Targeting young men: audience-centered communication for AIDS prevention in Vietnam. Monograph Series No. 4. Hanoi: CARE International in Vietnam; July, 1994.
3. McKenzie J, Smeltzer JL. Planning, implementing, and evaluating health promotion programs. 3rd ed. Needham Heights (MA): Allyn & Bacon; 2000.
4. Piotrow PT, Kincaid DL, Rimon JG, Rinehart W, editors. Health communications: lessons from family planning and reproductive health. Westport (CT): Greenwood Publishing Group; 1997.
5. Rogers EM. Diffusion of innovations. 4th ed. New York: Free Press; 1995.