The use of qualitative research methods as a component of efforts to assess and evaluate HIV prevention programming effectiveness has received much attention during the past decade. This chapter is intended for program managers who accept the idea that qualitative evaluation research can ultimately improve the quality of programming but who know very little about the methods involved. The information here can help guide program managers as they make decisions about resource allocation and evaluation demands.

Probably the most important piece of advice to those interested in using qualitative research for evaluation purposes is to identify local social scientists skilled in qualitative methods to assist with data collection activities. Manuals cannot replace technical expertise; they can only supplement or assist. Qualitative research skills are not confined to anthropologists or sociologists. Skilled researchers may be most easily identified by contacts with local research firms, university departments, or through donor agencies who have funded past research projects. Whether the researchers are native to the country or expatriates, they need to have the necessary local experience (geographically) and language experience to be able to function easily in short-term research environments. Conducting qualitative research is not any easier or less expensive than survey research, and the analysis effort is often more difficult and time consuming.

The chapters of this Handbook describe a variety of instruments or tools for evaluating HIV prevention programming, including standardized survey instruments for specific target groups. With qualitative evaluation research, however, it would be difficult to propose standardized tools. For example, at the beginning of an HIV epidemic and in the context of a hostile political climate in denial, what needs to be known about a target group that has not previously been reached and that is clandestine, stigmatized, and not well organized for advocacy is very different from what needs to be known in the midst of an advanced epidemic with high prevalence rates about a target group that has been receiving interventions for several years in a supportive political climate and that is organized for advocacy. These two extremes represent a long continuum. As a result, question topic guides that are used to gather information have to be tailored specifically to the target groups involved and to the particular information needs at a given time. These guides are as varied and unique as the languages and cultures of the target groups. Qualitative research presents reality in the words of the subjects of the research, and standardization is not the objective.

This chapter begins with some definitions and then briefly describes several of the more common qualitative research techniques that have been used in the context of evaluating HIV prevention programming in the recent past. The methods or techniques are considered tools for gathering evaluative qualitative data–tools to be used by people who have technical expertise in qualitative research.

Defining "Qualitative"

Because the word "qualitative" is an adjective, it is difficult to define without also describing the nouns the adjective frequently modifies:

• Qualitative data consist of words describing non-structured or semi-structured observations of reality, resulting in in-depth information of high validity about a relatively small number of cases (namely, respondents, interviewees, people) from the point of view of the people studied.

• Qualitative information provides insights into attitudes, beliefs, motives, and behaviors of target populations, including in-depth understanding about what people think and how they feel, using the actual words of the people being interviewed or observed.

• Qualitative methodologies attempt to answer the "why" questions and deal with the emotional and contextual aspects of response, adding "feel," "texture," and nuance to quantitative findings, again, from the point of view of the people being studied.

• Qualitative research does not attempt to generalize to larger populations because it usually involves smaller numbers of non-randomly selected respondents; however, the way that respondents are chosen could allow some degree of limited generalization.

Sometimes it is easier to define "qualitative research" by describing what it is not: qualitative research is any kind of research "that produces findings not arrived at by means of statistical procedures or other means of quantification"1; p.17. Some of the data may be quantified, but the analysis itself is qualitative. Or the data may be textual (qualitative) but may be coded in such a way that computerized content analysis might produce quantitative or statistical results. However, for our purposes, we uses the term "qualitative" to refer to nonmathematical analytical procedures resulting in findings derived from data gathered by a variety of means that might include observations, interview, and reviews of documents, books, videotapes, and even data that have been quantified for other purposes, such as census data1.

The information in this chapter focuses primarily on qualitative interviewing and observation techniques, and not on interpreting secondary data from written documents. The following descriptions of techniques (sometimes called approaches or methodologies) focus on those commonly used when designing or evaluating HIV/AIDS prevention programming.

Tools And Techniques For Collecting Qualitative Data

The complementary techniques described in detail below have been used by evaluation researchers to help assess changes in sexual behavior, knowledge about HIV/AIDS and STIs, beliefs about transmission and prevention, and attitudes toward various aspects of the epidemic. The concurrent use of qualitative research to complement knowledge, attitude, behavior, and practices (KABP) surveys provides a comprehensive, in-depth understanding of the context in which specific behaviors occur. In addition, formative qualitative research conducted at the beginning of interventions has contributed to solid project design and materials development. Complementing survey data, qualitative data help to track and assess, at key times, the shifts in trends in sexual behavior that will lead in the long term to reduced HIV incidence. The protocols and instruments for assessing capacity building of non-governmental organizations (NGOs) and government institutions also incorporate qualitative data collection techniques.

Participant Observation
The major source of qualitative data is what people say in interviews or write in documents. However, major differences often exist between what people say or write and what they actually do. Direct participation in and observation of the phenomenon of interest is likely to yield more accurate data. However, in HIV/AIDS prevention, the phenomenon of interest is sexual behavior, which puts certain limits on the level of participation of the researcher or evaluator. Tracking trends in sexual behavior change is a challenging task because of the sensitive subject matter and the private nature of the settings in which such behavior occurs. There are, however, other behaviors occurring in the wider context of sexual behavior in which a researcher can participate and observe.

Simply described, participant observation involves unstructured interviews (everyday conversation) and unstructured observation that are recorded in great detail in written field notes. Participant observation means that the researcher participates in the lives of the people under study while at the same time maintaining enough professional distance to allow adequate observation and recording of data2. In evaluating HIV interventions, participant observation means immersion to some degree in the setting in which interventions occur–physical presence in the environments in which high-risk sexual behavior is likely to occur or to be initiated. Participant observation produces textual descriptions (words, paragraphs, and pages) describing the daily life and events of the study population.

"Classic" participant observation, as defined by anthropologists, means that the ethnographer lives and works for 6 months to a year or more in the community under study, learning the language and viewing behavior patterns over time. Realistically, such a long-term approach to evaluation research is seldom possible. Therefore, participant observation in its strict traditional sense is not an approach commonly used in evaluating the outcome or impact of HIV prevention programming. Rapid ethnographic techniques that incorporate participant observation are often more practical and still extremely useful, particularly in formative or baseline research conducted to design intervention strategies.

Understanding the behavioral context of AIDS interventions involves focused description and systematic note-taking. To rapidly construct an accurate descriptive context, researchers must already be part of the culture, know the language, and have an established network of contacts or gatekeepers who can assist in putting together a description. Their application of the technique of participant observation at the design stage of an intervention, or even during a final evaluation, thus builds on their past participatory history with the target population and geographic location in question. In short, participant observation contributes to a more accurate (qualitative) context in which one can interpret the meaning of (quantitative) indicators of program achievement.

What exactly do participant observers do? Participant observation involves attention to all the elements that are normally needed to tell a story. Those elements include the following3:

• Who is present? How can you describe them? What roles are they playing? How did they become part of the group? Who directs the group?

• What is happening? What are people doing and saying? How are they behaving? How and what are they communicating? What body language are they using?

• When does this activity occur? How is it related to other activities or events? How long does it last? What makes it the right or wrong time for this to happen?

• Where is this happening? How important is the physical surrounding? Can and does this happen elsewhere?

• Why is this happening? What precipitated this event or interaction? Are there different perspectives on what is happening? What contributes to things happening in this manner?

• How is this activity organized? How are the elements of what is happening related? What rules or norms are evident? How does this activity or group relate to other aspects of the setting?

A participant observer, then, takes notes on all of these aspects during and after time spent in a setting, and develops the notes into a coherent written record of his or her experiences participating in and observing the context under study. Participant observation, conducted in a relatively rapid fashion, is useful to evaluators in many of the locations frequented by groups targeted for AIDS prevention.

Mapping
Creating maps of geographic areas in which research is conducted is an important complementary data collection technique frequently used at the beginning of participant observation and ethnographic studies. Maps can show major action settings, social divisions of a community, directions and distances between key sites, and natural features of the landscape4.

Occasionally, lists and maps of locations of key gathering points for target group members, such as brothels, bars, massage parlors, truck stops, hotels, or other locations, may exist Sources of sexually transmitted infection (STI) diagnosis and treatment services, condom sales outlets, or community distribution points for contraceptives are additional types of locations for which data may already exist. In most settings, however, there is likely to be no geographic orientation to key points of existing or potential interaction with target group members. In those cases, implementing agencies conducting formative research before intervention design or at the beginning of implementation should create maps that identify the locations important to the intervention.

For example, in a rapid assessment of the transport industry conducted in Papua New Guinea, researchers constructed 12 maps of major ports, truck stops, and several ships' interiors. The maps provided a way to visualize the spatial arrangements where sex could be negotiated and performed. They also showed the relationship between the locations of markets, trade stores, and other current or potential outlets for condoms or venues for educational activities and the movements of the transport workers5.

An intervention program in Bangkok, Thailand, used mapping to identify occupational networks and their leaders, including pilot sites and numbers of workers at each site, in six districts. The mapping process required 2 months and relied on both primary and secondary data. First, aerial maps of the districts were obtained through the design office of the local district administration. The maps with blank streets and structure markings were enlarged and systematically filled out using symbols to represent places of employment, low-income residences, and popular gathering sites. The local post office staff was particularly helpful in this phase, but ultimately, the mapping teams had to canvass the district on foot and by motorcycle taxi and other means to verify and update the map entries. The community network inventory for each district enumerated target sites and included information on number of workers per type of site6.

Individual In-depth/Key Informant Interviewing
Individual interviewing means that a single researcher interviews one other person, most often referred to as an "informant," which simply means "the individual who provides information." Other terms used include interviewee, respondent, source, participant, actor, consultant, or friend.

Key informants are respondents who have special knowledge, status, or access to observations denied to the researcher and who are willing to share their knowledge and skills. Because these types of informants tend to be especially observant, reflective, and articulate, they are usually consulted more than once or even regularly by the research team. Informants' abilities to describe events and actions may or may not feature analytical interpretation. In other words, some key informants may simply describe things without offering their thoughts on meaning or significance.

The process of interviewing can be highly structured, with a precoded questionnaire, or completely unstructured and open-ended. Something in between might be a one-page list of topics to be covered in an interview or a set of questions without precoded answers. Interviewers might record the interview and have a tape transcribed later, or they might take notes during the interview and then expand and formalize the notes just after the interview. Evaluative data collected through individual interviews also can be used to assess project implementers' opinions about projects' effect on communities, about the successes and challenges of implementation, about lessons learned, and about future directions interventions could or should take. "Project implementers" can include project managers at various levels, community outreach workers, peer health educators, trainers, decision makers, community leaders, and other categories of individuals connected with an intervention. Their opinions, observations, and interpretations add a qualitative component to quantitative process data that concentrates on people trained, people educated, condoms distributed and sold, and materials produced and distributed.

Question topic guides for these kinds of individual interviews can range from the most simple set of five or six open-ended questions that might take 20 minutes to administer, to a more complex and detailed topic guide of specific implementation-related questions. A general and open-ended set of simple questions could include the following:

• How do you feel members of this community have responded to this intervention? [wording could be specifically tailored to the intervention]

• What have been the strengths of this project from your perspective?

• What have been the weaknesses of this project and how could they be addressed in future activities?

• What are the lessons to be learned from this community's experience with the project that the implementing agency can use in future projects? How would you do things differently?

• How do you think this community would respond to a community-based condom distribution and sales program–in other words, do you think it would be a good idea to have condoms available for sale at the village level in this country? How do you think such a program should be implemented?

An end-of-project evaluation of an intervention with rural Muslim Ugandans incorporated these five questions to collect information from key individuals who had participated as Family AIDS Workers, trainers, or coordinators in the project. Six members of the evaluation team conducted four to five interviews each during 3 weeks of collecting survey data on a sample of the population within the target area. Each interviewer made notes on the comments from respondents, then summarized the notes from interviews into one document. Then the team worked together to discuss the various responses to each question in a group analysis exercise that resulted in a single document summarizing the results. Interviewers made a special effort to preserve short direct quotes from respondents in their notes so that the exact words of individuals could complement the summary statements made about the answers to the questions. Because the topic guide was very short and specific, the interviews did not last a long time and the analysis time was minimal. The questions were open-ended, allowing respondents to interpret the meaning of the questions in their own way7.

In another example, as part of the final evaluation of a country program in Haiti, evaluators conducted 42 individual interviews with a variety of project implementers, including project managers, community outreach workers, peer health educators, and other fieldworkers. The interviews focused on identifying "lessons learned" in the area of behavior change communication for HIV/AIDS prevention by collecting information on experiences from the field. Researchers conducted interviews over a 2-week period in either Creole, French, or English by two communication officers. Following that period, the interviews were transcribed, translated into English, and typed into a word processor, then imported into The Ethnograph software for analysis of textual data. Experiences described by the respondents illustrating principles, trends, and innovations were coded, analyzed, and reported8.

Focus Group Discussions
Focus groups involve a skilled moderator or facilitator who leads an informal but structured discussion or conducts an interview with a group of 6 to 12 people. An assistant moderator takes notes, operates the tape recorder, and deals with interruptions. Sometimes there are observers. Focus groups are used to gain an in-depth, but not necessarily representative, understanding of the attitudes, beliefs, and perceptions of a specific group of people in their own language. Reports of group discussions should contain a wealth of direct quotes from the participants that illustrate the summary points presented by the researchers.

The role of the moderator is critical: He or she should attempt to maintain the group discussion among the participants, and needs to be skilled in stimulating participation, guiding discussion, and probing for detail and depth without seeming to dominate the discussion. The moderator must also be able to manage overbearing participants and draw shy participants into the conversation9.

Focus group discussions can be used for types of research where there is a need to explore unknown aspects of experiences, feelings, and beliefs in such a way that the information is useful to researchers and program managers. Focus groups can offer an opportunity to probe people's perspectives on some set of topics, rather than assuming that the researcher already knows what the appropriate categories or issues are. The group setting is particularly useful because the moderator can listen to interaction between people and observe agreement, disagreement, and clarification. Focus groups generally are not working well when the "discussion" becomes a formal series of questions by the moderator and short answers by the participants.

These group discussions can be used in a variety of ways as components of evaluation research. They are commonly used in the planning stages of research and in the design of interventions. Focus group data can inform the content and language of survey questionnaires or can be used at the midpoint of a project to assess participants' feelings about how the project is going or where things need to change. Data from focus group discussions can be very useful to assist in explaining confusing findings or further clarifying the meaning of statistical results from KABP surveys. They are useful at the end of projects to assess project implementers' views about project successes, strengths, lessons learned, and how challenges were overcome. Focus groups also can be used at the end of projects to assess target group members' opinions about behavior change in their communities and about response to programs. Ideally they should be used along with other techniques and sources of data for cross-validating (triangulating) information.

A key characteristic of focus group discussions is that participants are not expected to reveal personal experiences. The emphasis should be on the participants' opinions about what "people like them" are doing or might do, rather than on the participants' personal behavior10. So, for example, instead of posing a question in a group discussion with university students, such as, "How many people here have ever used a condom?" the moderator might say, "In general, how do students at this university feel about using condoms?" The focus is on the participants' opinions about how other students like them feel about condom use. The discussion should reveal something about social norms at the university as well as the language students use to talk about sexual behavior and condoms.

Sensitive topics, such as sexuality and AIDS, can be quite difficult or relatively easy to discuss in focus groups, depending on the target group and the cultural setting. In Thailand, for example, young women workers and students were more comfortable talking about sexual behavior (their peers' or their own) in individual, open-ended interviews rather than in group settings. Sex workers and military personal, however, did not seem reluctant to discuss these sensitive issues in group settings. Other researchers in Thailand encountered no resistance at all in focus groups with males and females on the topic of male extramarital sexual behavior10. Pretesting focus group question topic guides can provide researchers with insight into how their target group members will react to certain lines of questioning.

Choosing participants can be tricky. The idea is to have a homogeneous group according to the characteristics important to the research questions. Generally age, gender, and status are key for any group. Participants should be of similar ages, the same gender, and similar status in terms of education, occupation, political status, or authority. In a focus group discussion with STI clinic staff, for example, it would be a mistake to combine counselors, nurses, and the clinic medical director in one group. If the focus group is to be with university students, it probably would not work nearly as well if the group combined sexually active individuals with students who have not yet become sexually active. A group convened to explore transport workers' attitudes about condom use would work better if men who used condoms frequently were gathered in one group and other men who never used condoms talked together in a second group. Mixing ethnic groups may or may not work well, depending on the cultural and linguistic situation. HIV serostatus may be an important characteristic, again depending on the objectives of the research.

The level of detail or instruction in the question topic guide depends on the amount of direction and information needed by the moderator. It is important to avoid recreating a survey questionnaire in a focus group topic guide. Even though a focus group topic guide may be designed to gain an understanding of people's perceptions of their peer's sexual behavior change, the wording of the questions will not be the same as when individuals are being interviewed about their own behavior. If the moderator is familiar with the research objectives and the target population, the question guide could simply be a list of topics with reminders about when to probe and key words needed to explore particular issues. Some topic guides may be as simple as five or six or a dozen questions, while others may be much longer. Again, the length depends on the research objectives involved; pretesting will determine whether the length of time is appropriate for the participants.

Focus group data are usually recorded on audiocassette tapes, which must be transcribed word for word and sometimes translated. The assistant moderator's notes are an important complement to the transcription because they contain descriptions of nonverbal communication during the session, information on the environment in which the discussion takes place (including timing, distractions, or other descriptive information), and information about the demographic characteristics of the participants. Analyzing these data is labor intensive and demanding of the principal investigators, whose understanding of the objectives of the research drives the analysis process. Computer-assisted data analysis is helpful for organizing and retrieving textual data, but preparing the text to be used by the software and coding the data are time consuming (this issue is discussed in greater detail later in this chapter). The quality of the analysis is enhanced by the use of an "overview grid," or matrix that summarizes a set of themes broken down by group, and also by the use of a team approach to interpreting the transcripts11. External sources of information on the issues involved, such as ethnographic material or survey results, should be used to check the plausibility of the focus group findings12.

Rapid Ethnographic Studies

"Ethnography is what those of us who are not anthropologists think anthropologists do. The popular image is captured by the vision of Margaret Mead in her tent, taking notes from the natives"13.

Traditional ethnography evolved over the past century as a way of describing exotic, remote, and culturally "different" people and environments. Although ethnographic research is no longer limited to "non-Western" cultures, the original process of ethnography remains that of discovering, describing, and analyzing a group's culture, way of life, or shared sense of reality within a specific geographic setting2,13-15. Ethnographers' priority methodology is participant observation conducted during intensive fieldwork. The findings are then interpreted from a cultural perspective–the assumption being that all human groups evolve a culture– and written up into a book or report16.

The concept of "rapid ethnography" evolved as a result of the need to make ethnographic data available to program managers in a timely fashion17,18. The methodology uses mainly qualitative (or anthropological) data-gathering techniques in a relatively short period of time (such as a few weeks) to describe a cultural setting, usually within specific demographic and geographic boundaries. In addition, an ethnographic perspective attempts to interpret the meanings behind observed behaviors, such as responses to disease14.

Rapid ethnographic research is particularly useful during formative evaluation stages. In Nepal, for example, a program design team drafted a plan for an HIV/AIDS prevention project focusing on at-risk behavior among commercial sex workers and their clients in the country's Terai Region, the southernmost districts adjacent to Nepal's border with India. Because no data existed on the sex industry in Nepal's international border areas, a rapid ethnographic assessment was conducted over a 3-month period to explore the social and behavioral context of commercial sexual interaction in five border cities. The results of the study helped refine the design of the overall intervention.

The assessment used qualitative and semi-quantitative methodologies for interviews with 56 female sex workers (FSWs), 100 clients, and 80 potential AIDS educators (health workers, education professionals, NGO staff, pharmacists, and social workers). These interviews were conducted by an assessment team consisting of an expatriate anthropologist with long-term experience in Nepal, a Nepalese physician with extensive experience in HIV/AIDS prevention programming, and male and female field interviewers. and clients of sex workers were relatively easy to obtain compared to interviews with the sex workers themselves. The sex industry in Nepal operated in an "underground" context; public officials underplayed the existence and size of the industry. Sex workers led a very mobile lifestyle and tended to be suspicious of researchers. Prostitution in Nepal varies widely by ethnicity, religion, caste, and degree of organization. Among the Badi ethnic group, prostitution is a social norm and respected profession, but among non-Badi women, persecution, transience, and isolation characterize the practice of the profession. The interviews revealed that levels of knowledge of HIV/AIDS, awareness, and access to and use of condoms differed between Badi and non-Badi sex workers, suggesting a need for different approaches to prevention.

The rapid assessment also provided better definition of the target population and helped focus the program design on a comprehensive, mutually reinforcing set of interventions (STI service improvement, condom distribution and use, targeted behavior change communication, and outreach education). The research revealed that the commercial sex industry operated in remote areas as well as urban centers, and provided new insights into the organization, lifestyle, and difficulties of Nepali sex workers. The range of variation among geographic locations and ethnic groups was surprisingly extensive.

Perhaps the most important finding emerged from a mapping exercise. During interviews with transport workers, each respondent was asked to draw on a road map of Nepal the preferred locations for commercial sex. The maps identified small rest stops along the major east-west and north-south transport routes leading from India into Nepal. Transport workers preferred remote, highway-based teashops, restaurants, and lodges as rest and entertainment sites to similar establishments in large urban locations. In Nepal, truck drivers and their assistants say they would rather rest and relax at small roadside locations where there is plenty of space to park and maintain their trucks and where police and others do not bother them as often.

Given men's preferences for remote, out-of-the-way locations and sex workers' identification with transient lifestyles, intervention planners determined that the Nepal program's geographical focus should expand beyond the major urban centers to include smaller commercial centers, transport bazaars, and small rest stops adjacent to Nepal's major highways.

As the Nepal strategy refined its focus to the nation's primary transport routes, other related factors were also considered, most importantly the initiation of new and large donor initiatives–the American Foundation for AIDS Research (amfAR) funding of 17 NGOs in Kathmandu Valley and the far eastern and western highways, and the European Community's plan to develop a public sector STI clinic targeting transport workers along the central region highway. These factors, combined with a more realistic analysis of available project financial, human, and NGO resources, refocused the program from a strategy covering the nation east-to-west, to one focused on the country's primary transport routes leading to and from India, all located in the Central Development Region–a more manageable area covering a road distance of 435 kilometers.

Targeted Intervention Research (TIR)
A specialized methodology for conducting rapid or focused ethnographic research, TIR has been used to help STI program managers understand their communities' perspectives on STIs and HIV/AIDS19,20. Like other types of ethnographic research, TIR uses mainly qualitative methods, but also can include a quantitative survey of a specific population, which allows evaluators to generalize findings to some extent.

TIR is designed to be used without outside technical assistance. It begins with the formation of a technical advisory group (TAG) that includes the STI program manager, someone responsible for the provision of STI services, a local communication expert, and a local social science researcher who supervises fieldwork. Led by the STI program manager, the TAG identifies the critical programmatic questions and concerns, then designs the details of the research, refining TIR methods to suit the specific local situation. The TIR is designed to be conducted over a period of 3 months, including a 1-month start-up, 1 month of data collection, and 1 month for data analysis and write-up.

The TIR methodology attempts to build consensus among TAG members about local programmatic priorities for STI control and to obtain information about the most common illnesses affecting adults, illnesses affecting the genital area, and illnesses transmitted through sexual intercourse. These descriptions of illnesses –"explanatory models"–include information on peoples' perceptions about transmission modes, causes and symptoms along with their timing and mode of onset, severity, and appropriate treatments. Explanatory models are constructed by combining information about an illness from the descriptions of many individuals, including community members and clinic workers, thus triangulating information to increase the validity of the data.

Analyzing Qualitative Data

"Analysis is just an old Greek word that means 'undoing'…Analysis is no more than the undoing of something already put together, something ready to be undone…Picking out things to know and to talk about in fact is the main way in which analysis has always been done…Analysis, the undoing of things, picking things apart, is a commonplace worldly procedure of selection"21.

Using Field Notes as the Foundation
Qualitative data come from fieldwork in which the researcher or evaluator spends time in the setting under study–whether the setting is a program, an organization, or a community22. Such settings for HIV prevention programming might include an STI clinic, a brothel, a truckstop, a factory, an antenatal clinic, a youth center, a women's group, a restaurant, a hotel, a roadside cafe, a university or secondary school, a family planning clinic, a theater, or a marketplace. Within these settings, researchers might participate and observe, talk to people or with groups, take voluminous notes, record conversations or discussions, review documents and records, or even videotape activities.

Field notes are to qualitative research what a filled-in set of questionnaires is to survey research. Field notes–description and direct quotations–are the source of the data that support the conclusions and interpretations that will ultimately be presented by the researchers. Even when using a tape recorder, the interviewer must take notes of key phrases, major points made by the respondent, and key words shown in quotation marks that capture the speaker's own language used. If, for some reason, a tape recorder cannot be used or if it malfunctions, it is even more critical to take detailed notes, especially of actual quotations as often as possible. It is perfectly acceptable to say to a respondent, "Could you repeat for me exactly what you said so I can write it down word for word. I don't want to lose that particular quote. Let me read back to you what I have written to make sure it is exactly what you said." Capturing what people say in their own words is the most important contribution of qualitative research to understanding human behavior.

The raw data of the field notes must then be reduced in quantity to a readable, narrative description organized into major themes and categories and illustrated with case examples and direct quotations from respondents. A summary of the results, either alone or in combination with quantitative results, should be presented to project stakeholders, disseminated with an emphasis on lessons learned, and referenced again during the design of future interventions. The remainder of this section describes the process used by researchers to distill raw field notes into usable findings. Increasingly, computers are used to help with this task and several important considerations in their use also are discussed.

Turning Data Into Usable Findings
Analyzing qualitative evaluation data should proceed under the direction of someone trained in qualitative data management. One good definition of "analysis" is Miles and Huberman's, which describes the process as "consisting of three concurrent flows of activity: data reduction, data display, and conclusion drawing/verification"23; p.10.

• Data reduction means summarizing or coding large amounts of text into smaller amounts of text, and it occurs continuously throughout the evaluation research process. It involves selecting, focusing, simplifying, abstracting, and transforming the raw data of field notes or transcriptions into typed summaries organized around themes or patterns based on the original objectives of the evaluation research. Data reduction continues to occur until a final report is written.

• Data display is defined as "an organized assembly of information" that allows conclusions to be drawn and actions to be taken23; p.11. Most frequently, qualitative data are displayed as narrative text, which tends to overload peoples' information-processing capabilities. Matrices, graphs, networks, and charts can present information in compact forms that make the data accessible to program managers.

• Conclusion drawing/verification refers to the process of deciding what things mean, noting themes, regularities, patterns, and explanations. Researchers will draw conclusions throughout the entire data collection exercise, but eventually this process becomes more explicit and firm at the point when the final report is written. Conclusions must also be verified as analysis proceeds. As researchers try to explain what the data mean, they should continually examine their explanations for plausibility and validity–do their explanations make sense within the context of the study? In evaluating HIV prevention programming, it might be possible to test the validity of conclusions during presentations of preliminary research findings and interpretations to project stakeholders and/or members of target populations. Researchers could incorporate this feedback into a final version of the written report.

During the conclusion drawing phase, the principal investigator conducting qualitative research has a very different role than he or she does in survey research. As Knodel explains:"In survey research, the coding of questionnaires is a routine matter of data processing to be relegated to assistants, and tabulations can be run by a programmer. In contrast, in focus-group research, code-mapping and overview grid construction are essential parts of the analysis process itself. It is exactly through these time-consuming and somewhat tedious tasks that the researcher comes to understand what the data are revealing. They can only be relegated to an assistant at the cost of detracting from the quality of the analysis…"12; p.103.

In developing a budget for focus group discussions, therefore, a sufficient amount of time must be included for analysis by the principal investigator. These three components of analysis–data reduction, data display, and conclusion drawing/ verification–need thorough documentation in the final reports of evaluation research so that readers can clearly understand how researchers arrived at certain conclusions. Only by understanding just what is done during data analysis of qualitative data can researchers delineate more reproducible methods.

Using Computers to Assist in Analyzing the Content of Qualitative Data
The biases and subjectivity inherent in qualitative data analysis, compounded by humans' limited ability to process large amounts of information, can be reduced somewhat by using computer software to assist in organizing, sorting, and categorizing textual data for a content analysis. These textual data can include literature, transcripts of discussion groups, notes from interviews, official reports, or any material that can be typed as text. People often associate computerized data analysis with quantitative or statistical approaches. This makes sense, given that computers are needed to efficiently process the calculations required for many statistical procedures and for large sample sizes. However, computers are also valuable tools for analyzing qualitative data. As with statistical analysis, rigorous qualitative analysis can be time consuming. Many of the basic procedures can be streamlined and enhanced through the appropriate use of computers and software designed for content analysis. Box 12-1 provides some suggestions on choosing a software package.

It is important to recognize that using a computer to conduct a content analysis of qualitative data does not somehow render the results "quantitative" or change the basic processes of data interpretation. The computer simply assists the researcher in coding and analyzing the data. Coding data refers to the process of associating a code word with pieces of the text that represent a certain theme. This allows the analyst to search for the codes rather than the words in the text. Using a computer to assist in this way with analysis still requires skill in qualitative data analysis and some expertise in the subject area. Moreover, planning the process is crucial: The final analysis will only be as sound as the logic that grounds the research. Computers cannot interpret the findings. As with statistical analysis, it is incumbent on the researcher to carefully interpret the findings, place the analysis in context, and relate findings to theory.

Although computer-organized text may improve the reliability and validity of the analysis, there can be trade-offs in terms of the time and effort required to complete the analysis and final reports. Evaluation activities usually need to produce results quickly to meet various deadlines, so researchers should make a realistic assessment of timing and budget resources before using computer software in qualitative data analysis. The following sections review some of the basic issues involved in using computers to analyze qualitative data.

Advantages and Disadvantages of Using Computers for Content Analysis
Using computers to assist in analyzing qualitative data is advantageous because it allows evaluators to:

• be very systematic with the data analysis;
• search data according to predefined codes and combinations of codes;
• use the coding from a second person who analyzed the data to identify biases in coding; and
• manage large amounts of text and associated codings.

Using computers to analyze qualitative data also has some disadvantages because:

• it is time consuming to set up and code the data;
• it may be necessary for evaluators to learn to use a computer and software;
• the computer hardware can malfunction;
• data can be lost due to computer failures;
• it can be time consuming to use a computer; and
• the computer and software are expensive.

Most of the hardware and software disadvantages can be easily dealt with through planning, training, and patience. Disadvantages related to cost and time must be weighed against the advantages of using computers for analysis. For many small projects, especially if the computer approach will never be used again, it is best to conduct an analysis manually.

Steps Involved in Using a Computer to Code and Analyze Data
Several computer programs are available for use in coding and analyzing data, and all are different. However, using most of the commercially available software packages to assist in conducting content analysis involves a few basic steps. These include:

• transcription (and sometimes translation);
• formatting;
• coding data; and
• interpreting data.

Once the first piece of data has been collected (one group discussion or a single interview), it can be transcribed into computer-readable format (usually ASCII) to be used by the computer program. Occasionally, it will be necessary to translate the text into another language. Translation should be undertaken carefully–normally with the text translated twice for comparison, or back translated (language to English, French, or Spanish, and back to language) to detect differences in translations. The data are then formatted to be used with the particular software program. Next, the data are systematically coded using a predetermined coding scheme. Note that the coding scheme can, and should, be modified as the coding progresses. Normally, the evaluation team starts with a basic set of issues to code and, as the process of reading the text progresses, they will add new codes to the list based on their reading. Once the data are coded, the software are used together with the coded data to produce interpretations and conclusions.

It is probably best to have two people code the data, using the same set of code words, and comparing the findings for discrepancies. When discrepancies are found, a system should be in place for resolving them–such as having a third person also read and code the text.

Controlling for Bias in the Data or in Interpreting Data
It is important that findings not be biased by the personal opinions of the researcher. Rather, research findings should be based on an objective interpretation of the data. However, because so much is open to interpretation, it can be difficult to analyze qualitative data objectively. Using computers to assist in analyzing qualitative data can help reduce bias and identify bias when it does occur. If a systematic plan of analysis is followed, there is less chance for bias in interpreting results. For example, an analyst who finds that a text has few coded responses to substantiate a particular interpretation of the content should wonder whether the coding is truly unbiased. Or, if a second person who codes the same data has different coding patterns (in other words, codes the same passages differently), the analyst should be especially careful in interpreting those particular findings.

Triangulation And The Role Of Qualitative Research

The concept of triangulation is taken from land surveying–a person being able to identify his or her location on a map by taking bearings on two different landmarks and determining their intersection. The two landmarks and the surveyor become the three points of a triangle. In evaluation research, triangulation can be of four different types22:

• data triangulation–using several data sources in a study;
• investigator triangulation–using several different researchers or evaluators;
• theory triangulation–using multiple perspectives to interpret a single set of data; and
• methodological triangulation–using multiple methods to study a single problem or program.

While triangulation is ideal, it is also expensive and poses problems for evaluators' limited budgets, short time frames, and political realities. Nevertheless, most evaluation experts agree that triangulation greatly reduces systematic bias in the data. "Triangulation is a process by which the researcher can guard against the accusation that a study's findings are simply an artifact of a single method, a single source, or a single investigator's biases"22; p.470.

In assessing and measuring behavior change resulting from interventions, KABP survey data and qualitative focus group data have most commonly been used to compare and contrast perceptions of trends in behavior change as program managers face the task of evaluating their program achievements. Assessing experiences from the field–"lessons learned"–can involve group interviews with peer health educators and individual interviews with project managers. These qualitative sources of information, contrasted and compared with survey data on common behavioral indicators, provide multiple methods as well as multiple interpreters as multidisciplinary teams prepare final project reports.

Interpreting, Reporting, And Disseminating Data Results

Because the purpose of evaluation research is to make a difference in decision making and programmatic action, a researcher is obliged to report and present the results of his or her data reduction efforts and to share interpretations of the findings with project stakeholders. Boxes 12-2 and 12-3 provide guidance on writing qualitative research reports. Ideally, the evaluator provides stakeholders with description and analysis, verbally and informally, and includes the stakeholders' reactions as part of the data in the written report. Stakeholders expect evaluators to "confirm what they know that is supported by data, to disabuse them of misconceptions, and to illuminate important things that they didn't know but should know"22; p.423.

The definition of the word "interpret" is "to clarify the meaning of by explaining or restating." This means that the researcher "clarifies the meaning" of the observed behavior or descriptions (data) by restating the observations made or the things that respondents have told the interviewer. The evaluation researcher uses his or her training and expertise to explain and clarify the data for people who have not participated in the data collection experience. "Interpretation" is really an art that requires sophistication, maturity, and wisdom. "Interpretation" means explaining what is happening, whether the events are observed first-hand or described by someone else. Interpretation is part of analysis and goes beyond data description and data reduction.

When the evaluation researcher explains and interprets the meaning of information, he or she is doing so from a certain personal perspective and in a certain context. That perspective and its context should be identified at the beginning of the written report in order to enhance the readers' abilities to understand the research results.

Conclusion

During the past decade, much attention has been devoted to the use of qualitative research methods as a component of efforts to assess and evaluate the effectiveness of HIV programs. Today, there is consensus that qualitative data and research methodologies are a valuable complement to quantitative data and research techniques, and in fact, that these research approaches should be used in combination to assess the progress and effects of prevention and care interventions.

Several methodologies are commonly used in qualitative research, including participant observation, mapping, in-depth interviews, focus group discussions, and rapid ethnographic studies. These approaches are useful in all types of evaluation research, from formative evaluation that is used in planning and designing interventions, to process evaluation that tracks how well interventions are carried out, to effectiveness evaluation that assesses the impact and outcomes of interventions.

By examining issues from the point of view of the people being studied, qualitative research provides context and meaning, often answering the "why" question of behavior and barriers to behavior change. Answers to these questions permit programs to develop more effective communication messages and design better services that respond to target groups' barriers and needs.

Vignettes and the real-life experiences of individuals captured in qualitative data collection can serve as an effective evaluation tool for informing the decisions of program managers and policymakers. Far from the often dry statistics of quantitative data, the personal details of human experience can captivate an audience so that they can better understand and grasp the effects of a program on people.

Because the raw data of qualitative research consist of words, thoughts, opinions, quotes, and observations, the process of content analysis is often a labor-intensive and time-consuming activity. Computer software can be helpful in organizing the raw data and streamlining and enhancing the analytic process, but it has limitations. It is incumbent on the researchers to carefully interpret the findings, place the analysis in context, relate the findings to theory, and clearly present the results to stakeholders. The process of analysis and reporting must be closely guided by principal investigators who are trained in qualitative data management and experts in the subject matter.

References

1. Strauss A and Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park (CA): Sage Publications; 1990.
2. Fetterman DM. Ethnography step by step. Vol. 17, Applied Social Research Methods Series. Newbury Park (CA): Sage Publications; 1989.
3. Bogdewic S. Participant observation. In: Crabtree BF, Miller WL, editors. Doing qualitative research. Vol. 3, Research Methods for Primary Care. Newbury Park (CA): Sage Publications; 1992. p. 45-69.
4. Pelto PJ, Pelto GH. Anthropological research: the structure of inquiry. London: Cambridge University Press; 1978.
5. Jenkins C. Final report: behavioral risk assessment for HIV/AIDS among workers in the transport industry, Papua New Guinea. PNG Institute of Medical Research, Goroka. Arlington (VA): Family Health International; 1994.
6. Supammatas S. Community mobilization working paper #1, the community network diagnosis. Faculty of Public Health, Mahidol University. Durham (NC): Family Health International/AIDSCAP; 1993.
7. Kagimu M, Marum E, Wabwire-Mangen E, et al. 1995 Family AIDS education and prevention through Imams (FAEPTI) project, follow-up evaluation report. Islamic Medical Association of Uganda (IMAU). Kampala: World Learning Inc.; March 1995.
8. Damier E, Mahler H. BCC experiences from the field in Haiti. Arlington (VA): Family Health International; 1996.
9. Shedlin MG, Schreiber GM. Using focus groups in drug abuse and HIV/AIDS research. Paper prepared for the National Institute on Drug Abuse (NIDA) Technical Review: qualitative methods in drug abuse and HIV research. Rockville (MD): NIDA; 1994.
10. VanLandingham M, Knodel J, Saengtienchai C, Pramualratana A. Aren't sexual issues supposed to be sensitive? Health Transit Rev 1994;4(1):85-90.
11. Jato M, van der Straten A, Kumah OM, Tsitsol L. Using focus-group discussions to explore the role of women's groups (tontines) in family-planning information dissemination in Yaoundé, Cameroon. Health Transit Rev 1994;4(1):90-95.
12. Knodel J. Conducting comparative focus-group research: cautionary comments from a coordinator. Health Transit Rev 1994;4(1):99-104.
13. Gilchrist VJ. Key informant interviews. In: Crabtree BF, Miller WL, editors. Doing qualitative research. Vol. 3, Research methods for primary care. Newbury Park (CA): Sage Publications; 1992. p. 70-89.
14. Schensul JJ, Schensul SL. Ethnographic evaluation of AIDS prevention programs: better data for better programs. In: Leviton LC, Hegedus AM, Kubrin, editors. Evaluating AIDS prevention: contributions of multiple disciplines. San Francisco: Jossey-Bass; 1990. p. 51-62.
15. Vidich AJ, Lyman SM. Qualitative methods: their history in sociology and anthropology. In: Denzin NC, Lincoln YS, editors. Handbook of qualitative research. Thousand Oaks (CA): Sage Publications; 1994. p. 23-59.
16. Wolcott HF. Writing up qualitative research. Thousand Oaks (CA): Sage Publications; 1990.
17. Bentley ME, Pelto GH, Straus WL, et al. Rapid ethnographic assessment: applications in a diarrhea management program. Soc Sci Med 1988; 27(1):107-116.
18. Scrimshaw SCM, Carballo M, Ramos L, Blair BA. The AIDS rapid anthropological assessment procedures: a tool for health education planning and evaluation. Health Educ Q 1991;18(1):111-123.
19. Helitzer-Allen DL, Allen HA. The manual for targeted intervention research (TIR) on sexually transmitted illnesses with community members. Arlington (VA): Family Health International/AIDSCAP; 1994.
20. Helitzer-Allen DL, Allen H, Field ML, Dallabetta G. Targeted intervention research on sexually transmitted illnesses. Pract Anthropol 1996;18(3):20-23.
21. Rose E. The first book of the werald: the world undone. Boulder (CO): The Waiting Room Press; 1991. Cited in: Seidel J, Friese S, Leonard DC. The Ethnograph v4.0: a user's guide. Amherst (MA): Qualis Research Associates; 1995.
22. Patton MQ, editor. Qualitative evaluation and research methods. 2nd edition. Newbury Park (CA): Sage Publications; 1990.
23. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd edition. Thousand Oaks (CA): Sage Publications; 1994.
24. Weitzman EA, Miles MB. Computer programs for qualitative data analysis. Thousand Oaks (CA): Sage Publications; 1994.
25. Miles MB, Weitzman EA. Choosing computer programs for qualitative data analysis. In: Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. 2nd edition. Thousand Oaks (CA): Sage Publications; 1994. p. 311-317.