FHI - Chapter 14

STD Program Monitoring and Evaluation
Preview Chapter 14 Introduction This chapter discusses the role monitoring and evaluation plays in STD/HIV control. The monitoring phase tracks and counts events, activities, people and objects. It includes either periodic or continuous data collection. Evaluation documents the magnitude of STDs, including HIV, and the positive impact of control measures, information that can be used to convince policy makers groups of the value of control programs. Setting Goals and Objectives Defining goals and objectives is the first step in monitoring and evaluating an STD control program. A goal refers to the long-range program activity, while an objective is usually a statement of shorter-term accomplishments. Data Collection and Analysis: Monitoring The monitoring and evaluation process should measure parameters that describe what an STD control program is actually doing. These include measures to assess service delivery, staff performance, adequacy of staffing patterns, client satisfaction and response, and resource needs and allocation. Service Delivery Examples of monitoring parameters for service delivery include the number of patient visits (per day, week, month); number of condoms distributed; and the number of cases of specific syndromes seen. Staff Performance Monitoring staff and personnel-related factors involves the following components: Determining whether the staff-to-patient ratio is appropriate to the patient load Ensuring that staff receive both initial and ongoing in-service training Ensuring that staff receive supervision and feedback Client Satisfaction and Response A complete record system and routine reports give staff and managers information on the number of patients using STD services; the types of patients (sex, age group, etc.) coming in; the number of initial patient visits compared with the number of repeat visits; and the proportion of patients using the facility as their first treatment option. Resource Needs and AllocationRoutine monitoring of costs allows an STD program to pace itself, stay within budget, and assess efficiency and cost-effectiveness. Data Collection and Analysis For Evaluating ProgramsA periodic evaluation can focus on one particular objective at a time or attempt to assess progress toward all objectives and goals. The most common methods of data collection for evaluating the biological effectiveness of STD control programs are the following: Surveillance data from the STD control program STD-related morbidity indicators collected from non-STD service sites Results of special studies conducted either by the STD control program itself or by others Case Study Examples Two examples from Thailand and Rwanda show how relatively few pieces of information can be combined to help programs begin to understand their impact on STD transmission. A simple question-evaluation process can give some highly useful answers. Use of Information and Data Focusing clearly and at an early stage on how data will be used helps a program in a number of ways. For example, it helps it avoid collecting irrelevant data, sustain the flow of information and ensure that findings are used to improve the STD program as it evolves. Conclusion As an integral part of an STD control program, a successful monitoring and evaluation system is used as a basis for documenting a program's value and for programmatic improvement. Introduction Monitoring and evaluation are critical to the success of any STD control program. The monitoring phase tracks and counts events, activities, people and objects and can consist of either periodic or continuous data collection. The information gathered in this process, such as new trends and strengths and weaknesses of the program, should be used to improve programs. The evaluation phase periodically measures and analyzes progress toward attainment of stated goals and objectives. This analysis may be based on existing monitoring data that is aggregated or otherwise manipulated, or on a separate, unique data collection system. In addition, by documenting the magnitude of STDs (including HIV) and their consequences, and by showing the positive impact of control measures, evaluation can help program managers convince governments and funding agencies that STD/HIV control programs are a cost-effective use of scarce resources. The purpose of monitoring and evaluation is to ensure appropriate and high-quality services. Therefore, monitoring and evaluation should be an integral part of the design, development and implementation of STD control programs, not just a periodic activity linked to broader planning or funding cycles. The more integrated the evaluation system, the more successful it will be, and the less likely practitioners will lose interest in it or find it unduly burdensome. Interpreting evaluation data must always be done cautiously. Biases may be inherent in the data collected, the methods used for its collection and the sites included in the data collection process. A common example of bias is found in evaluating reported cases of STDs, where the patient population, type of STDs and number of patients seen may differ widely between the private sector and public clinics, which usually are the source of most case reports. Because cases are often grossly under reported by private clinicians, the overall number of patients seeking care for STDs in the private sector may be underestimated. As a result of such biased information, important service areas may be left out of either an STD control program or its monitoring and evaluation system. Setting Goals and Objectives The first step in monitoring and evaluating STD control programs is to define the goals and objectives of the program. A goal can be defined as the long-range purpose for all program activity, while an objective is usually statement of intermediate (shorter-term) and more specific accomplishment. For example, the goal of an STD control program may be "To reduce the morbidity and mortality from STDs in population X" or "To reduce the incidence of genital ulcer diseases by 25 percent within five years." Many factors will contribute to a goal, and a number of different objectives will need to be defined in support of it. The long-range goal of the program should be stated explicitly to help individuals at all levels of the program understand the ultimate rationale for their activities and how the intermediate objectives contribute to the realization of those goals. The objectives in an STD program should reflect both operational and biomedical efforts or activities. For example, an operational objective might relate to increasing the number of STD patients evaluated, treated and counseled about STD and HIV risk-reduction behavior in a given time period. A biomedical objective might be increasing the percentage of patients given the correct treatment regimen for urethral discharge and then cured of the infection. Goals and objectives must be meaningful and realistic. Otherwise, program staff and managers will become discouraged. STD prevention and control programs at different stages of development or experience should have different goals and objectives specifically tailored to their situation and resources. Shorter-term objectives allow progress to be recognized and celebrated and can be useful in motivating personnel at all levels to continue improving their performance. Periodic reviews of all goals and objectives should be scheduled to ensure their continued feasibility and relevance to the overall program and to the ever-changing environment in which STD control programs operate. Furthermore, program goals and objectives should be set collaboratively by all parties involved and should not be imposed by outside evaluators. Encouraging input from STD program staff at many levels will increase compliance with data collection plans. It will also help ensure that the data selected to measure progress toward a given goal or objective fit the capabilities of the facility. For example, service sites with no laboratory services may file monthly reports of syndrome-based diagnoses, treatments used and patient demographics rather than reports based on laboratory results. Potential interpretation problems should be considered when objectives and goals are set so that biases can be anticipated. The presence of bias does not invalidate a measure if the amount of error can be estimated. For example, laboratory reporting of positive rapid plasma reagins (RPRs) is a very accurate, mechanism to estimate syphilis prevalence. But the reports may represent old, new and even treated cases of syphilis. New cases would be very difficult to distinguish from the others. However, over the long term, trends in new or incident cases of syphilis can be estimated by measuring syphilis prevalence in new, sexually active individuals (persons under the age of 20, for example) or by the proportion of specimens that demonstrate RPR titers greater than 1:8. The process of setting goals and objectives can be greatly facilitated by a comprehensive baseline assessment of the physical and human resources and actual services available at the various STD control program facilities. Examples of parameters to assess include the following: Whether prevention information is part of the patient/provider interaction What types of patients currently utilize the STD services (sex, age, marital status, etc.) Existing logistical and managerial support systems The nature and scope of the existing data collection mechanisms Simple features such as the average patient waiting time, number of patients per day, number and type of staff at the facility at all times, availability of antibiotics and condoms, and adequacy of water, lighting and cleanliness also are essential to providing good services and should not be overlooked. This type of baseline assessment will help program managers identify critical areas in need of improvement, help determine priorities and set realistic objectives and goals for an STD control program. For example, if a particular facility identifies chronic shortages of drugs and condoms or insufficient examination space for the current patient load, the program's first objective could be correcting these problems. Finally, determining what information will be most useful and affordable to collect is key to the sustain-ability of monitoring and evaluation efforts. The quality and quantity of the data, together with the cost and feasibility of obtaining the data, should be weighed when planning the STD control program's monitoring and evaluation activities. It may be more important, particularly in the early phases of program upgrading, to focus on a small number of easily collected pieces of data rather than on trying to implement a complex system of sophisticated measures that may collapse under the weight of reporting requirements. A good rule of thumb is to collect only what is needed and to use what is collected. Data Collection and Analysis: Monitoring The monitoring and evaluation process should measure parameters that describe what an STD control program is actually doing. These include measures to assess service delivery, staff performance, adequacy of staffing patterns, client satisfaction and response, and resource needs and allocation. Current clinic records can and should provide useful parameters and are one of the best places to start defining data collection options. Service Delivery Some examples of monitoring parameters for service delivery include the following: the number of patient visits (per day, week, month); the number of condoms distributed; and the number of cases of specific syndromes seen (e.g., urethral discharge, genital ulcer or vaginal discharge). Such information can come from any service delivery source: public STD clinics, primary-care facilities, private-sector health-care providers or hospitals. Surveillance systems and the reporting of specific diseases or syndromes is a special type of monitoring that can often include laboratory facilities as a data source (for syphilis serology, for example) and is addressed at length in Chapter 15 of this manual. How might some of the examples above be useful to an STD control program? The number of people who seek treatment for STDs and the number of cases of a specific syndrome diagnosed during a specified time period (e.g., a month), including breakdowns by sex, age and ethnic group, (if applicable), can help managers begin to understand the scope of the STD problem in a particular area. This is particularly true if such data are collected over time. Such information can also help managers anticipate changing personnel and supply needs. The number of condoms distributed or sold through service delivery systems during a specific time period will inform logistics and supply systems if their services are adequate, as well as track, in general terms, the effectiveness of basic prevention efforts. Two more complex pieces of information that can and should be collected as part of periodic monitoring are directly related to the quality of the care provided. They are "Prevention Indicators" (PIs) numbers 6 and 7 of the World Health Organization/Global Programme on AIDS (WHO/GPA).^1,2 These indicators measure the following: PI 6: The proportion of clients presenting for STD diagnosis and treatment who are treated according to national guidelines. Obtaining this information requires a combination of some of the routine data collection described above and direct observation to assess the correct use of the treatment protocol. PI 7: The proportion of clients presenting for STD diagnosis and treatment who receive appropriate prevention-related services. Special data collection (observation of the provider) is also required to obtain this information. It has proven difficult in some settings to strictly adhere to the GPA protocol. Provider and client interviews have sometimes been used in lieu of the observation component. While the use of interviews alters the validity and comparability of the data relative to the GPA protocol and may introduce additional bias, it may be the only option possible, particularly when dealing with private practitioners. Nonetheless, if practitioners use interviewing as the alternate technique of choice, comparisons can be made within a country, region or program. The following figures show results from two facility-based assessments conducted in Jamaica in 1991³ and in Malawi in 1994⁴. The Jamaica data were collected as part of early testing of the GPA protocols and will serve as the baseline for subsequent program improvements and evaluation. The Malawi data were specifically collected as baseline information for planned upgrading of selected sites within the STD service-delivery system. The "pilot and non-pilot" sub-groups in two of the graphs define whether the facility was participating in a National AIDS Control Programme (NACP)-implemented STD service upgrading process. The data collected in this assessment process provide very practical guidelines for programs, defining in a detailed manner the strengths and weaknesses of the facilities and personnel within the service-delivery system. The parameters that are lowest may be targeted by a program for special and intensive correction through procurement of supplies and commodities or staff education and training. Staff Performance Monitoring staff and personnel-related factors includes observation and analysis of a number of components: Determining whether the number and utilization of staff is appropriate to the patient load (e.g., time spent per patient) Ensuring that adequate initial training and appropriate refresher or in-service training is provided to all staff members not only as a means of assessing their skills and informing them of new protocols but also as a way to update staff on progress toward program objectives and goals Ensuring that appropriate on-site and external supervision and feedback is provided to all staff Supervisors should structure training efforts based on the information obtained from periodic assessments of personnel performance and facility capabilities (see Chapter 6). Client Satisfaction and Response In any health-delivery system, the staff is only half the picture. The other half is the client, or patient, population. Monitoring of patients relies most heavily on a complete, clear patient record system and simple, routine reports. These reports will give program staff and managers the following, pertinent information: The number of patients using STD services and, when the STD services are part of a primary health care or comprehensive clinic, what proportion of the patient load is attributable to STDs. The types of patients using STD services. Does one sex or age group predominate? Are particular sub-populations present in the patient base? For example, do members of the uniformed services use the clinic services? To what degree? The proportion of patients making initial clinic visits compared with repeat or return visits. This information can help program managers qualitatively assess the effectiveness of treatment and prevention counseling. The proportion of clients seeking treatment at the facility as a first option, either independently or by referral. This parameter can assist program managers in assessing treatment-seeking behaviors, partner notification and other referral efforts. Resource Needs and Allocation There are never enough financial resources to implement the perfect program. Routine monitoring of operating costs, such as salaries, travel expenses, and supplies and drugs, as well as new capital costs, such as remodeling and equipment purchases, is a very important part of program monitoring. Tracking these costs allows an STD program to pace itself, stay within budget, and assess efficiency and cost-effectiveness over time. Data Collection and Analysis: Evaluation A periodic evaluation can focus on one particular objective at a time or attempt to assess overall progress toward all objectives and goals. The specific factors assessed for an evaluation are dependent upon the goals and objectives of the program. For example, if a program's goal is to reduce the prevalence of urethral discharge or gonorrhea, the objectives leading to that goal would include an increase in condom distribution and partner referrals. The program evaluation would assess progress toward achieving those objectives over time and then compare those gains with the prevalence of urethral discharge or gonorrhea during the same time frame. Understanding that many factors, both internal and external to the STD control program, affect the prevalence of any disease or condition is, of course, critical to the interpretation of evaluation findings. The most common methods of data collection for evaluating the biological effectiveness of STD control programs are the following: Surveillance data from the STD control program STD-related morbidity indicators collected from non-STD service sites Results of special studies conducted either by the internal STD control program or by an outside group The advantages and disadvantages of each type of data collection will be described in the sections that follow. STD Surveillance Systems A surveillance system is a mechanism by which regular, standard reports of specified diseases, syndromes, pathogens or treatments are provided to a central authority for compilation and analysis. Chapter 15 provides detailed information on surveillance systems. Non-STD Service-Dependent Data Sources One of the most valuable sources of data for evaluating the success of an STD control program is derived from antenatal care services. In settings where use of antenatal services is high, serologic screening and treatment of antenatal patients for syphilis meet two important health objectives: It allows for treatment of syphilis in women and prevention of congenital syphilis (see Chapter 9). It provides the STD program with a relatively unbiased estimate of the prevalence of at least one STD in the sexually active population (see Chapter 15). Reporting of certain other diseases or conditions, including cases of congenital syphilis, gonococcal ophthalmia neonatorum and pelvic inflammatory disease (PID), may also be particularly useful for the evaluation of STD control efforts. The prevalence of these serious sequelae of STDs helps define the spread of STDs in the population. However, the diagnosis of these conditions is often difficult and costly and should not be undertaken lightly. Special Studies Special studies are used to answer certain questions or test a specific hypothesis. They are particularly useful for evaluation when they confirm information collected in a less controlled manner or provide information that will aid in the interpretation of routinely collected data. These studies are designed and implemented to answer specific questions in one or more designated sites following a standardized protocol over a set period or time. Examples of such special studies include surveys of patients, or the broader population as in the periodic national Demographic Health Surveys (DHS), on risk behaviors or use of or satisfaction with services, assessments of treatment compliance or partner referral, and laboratory studies to ascertain resistance patterns of prevalent STDs (see Chapter 12). Because so many factors can affect STD transmission and the reported data, the evaluation process should try to combine several different sources and types of data. By using a variety of methods and data, the evaluation process will provide a balanced, less biased measure of STD control program effectiveness. For example, a program should compare data on the prevalence of syphilis from both clinic records and surveillance or special serostudies instead of just relying on one source. Case Study Examples Two examples from Thailand and Rwanda show how relatively few pieces of information can be combined to help programs begin to understand the effect they are having on STD transmission. In Thailand, where prevention activities have focused on increasing condom use and access to STD treatment by commercial sex workers (CSWs), cases of STD in males in private sector clinics fell from over 237,000 in 1987 to approximately 38,800 in 1993. In this same time frame, reported condom use by CSWs rose from less than 20 percent of sex acts to 94 percent in June 1993, with a concomitant drop in new STD infections of 54 percent.⁵ A similar phenomenon has been reported in Rwanda by the NACP, where they examined rates of urethritis, as well as condom distribution, to military recruits.⁶ While incidence of urethritis had been approximately 10 percent in 1985 and 1986, once a condom promotion and distribution program began and reached a distribution level of approximately 5 condoms per month per recruit at the end of 1987, the incidence of urethritis dropped to approximately 2 percent. While neither of these examples can directly attribute the observed changes in the incidence of STD to the interventions, the data are strongly suggestive of the effect that prevention activities can have upon rates of STD. Furthermore, while all STD programs seek to reduce the rates of STD, the time frame, technology, personnel and geographic or population boundaries for control activities are specific to each situation. For example, an STD control program for the small island nation of Sri Lanka might have the same goal as a program in its large neighbor, India: "reducing the rate of syphilis to less than 2 percent nationwide." In each country, however, the programs would use vastly different strategies and methods to achieve this goal and evaluate their progress. Sri Lanka may be able to measure its progress through an active "universal" reporting system, where each case of syphilis in the entire country is counted. India, however, may rely on data collected only at selected sites or in populations serologically tested at regular intervals to determine success. An example of a simple evaluation process for a single clinical setting would be one that answered the following questions: How many cases of the various STDs were reported during this period? How does that number compare to the cases reported during the last period as well as comparable periods in previous years? Do the comparison reveal any trends? Are there specific factors that have caused this trend, such as changes in personnel, clinical hours, user fees, clinical definition or counseling techniques, or seasonal or social changes? Are these syndrome or disease trends different when broken down by age, sex, social status, special group or geographic area? If any of the STD diagnoses were confirmed in the laboratory, was the laboratory diagnosis consistent with the clinical diagnosis? Were there any significant laboratory trends, such as rates of resistant gonorrhea? Do these trends correlate with an increase in treatment failures? Use of Information and Data Information feedback mechanisms should be established at the same time as monitoring and evaluation parameters and data collection methods are defined. Feedback mechanisms can be as simple as routine monthly communication from a supervisor or manager to the clinic staff about the most recent number of patients treated and the trends in STD at the facility, or as sophisticated as annual, formal reports on all aspects of program performance. Focusing clearly and at an early stage on how data will be used helps a program in the following ways: By not collecting data that are of no direct benefit to the goals of the program By keeping information flowing in both directions By improving the quality of the data collected By ensuring that the findings of the monitoring and evaluation efforts are used to improve the program By demonstrating the progress and effectiveness of the program to improve internal morale and positively influence external audiences involved in funding and expansion The following table categorizes examples of the data collection and evaluation tool options for STD control programs, including indications of relative difficulty and cost. This information is provided to assist program managers and others as they determine the type of data needed or desired and the fiscal and human resources available in a program for evaluation efforts. Conclusion Monitoring and evaluation of an STD control program is a process used to ensure the quality and appropriateness of services. The first step is to define the goals and objectives of the program. Goals must be meaningful yet achievable. Getting input from many levels of the organization will increase compliance with data collection. Problems of interpretation and sources of bias should always be considered. A comprehensive baseline assessment may be helpful when setting goals and objectives. Monitoring tracks, counts and measures what an STD control program is actually doing. It can be either periodic or continuous. Surveillance is an important component of monitoring. It is a mechanism by which regular, standard reports of specified diseases, syndromes or pathogens are provided to a central authority for analysis. Monitoring of financial resources is a very important part of monitoring activities. An evaluation periodically measures and analyzes progress toward attainment of stated goals and objectives. It summarizes the monitoring information, interprets any developing trends and compares it to expectations. Feedback mechanisms can help ensure that information obtained from monitoring and evaluation is used to improve the program. As an integral part of an STD control program, a successful monitoring and evaluation system is used as a basis for programmatic improvement and to document a program's value. References Evaluation of a national programme: a methods package 1. Prevention of HIV infection. Geneva: World Health Organization, 1994. Mertens T, Carael M, Sato P, et al. Prevention indicators for evaluating the progress of national AIDS programmes. AIDS 1994;8(suppl 1):S359-S369. Bryce J, Vernon A, Braithwaite AR, et al. The quality of sexually transmitted disease services in Jamaica: evaluation of a clinic based approach. Bull WHO 1994;72:239—247. Chilongozi DA, Costello Daly C, Franco L, et al. Sexually transmitted diseases: a survey of case management in Malawi. Int J AIDS STD (in press). Hanenberg R, Rojanapithayakorn W, Kunasol P, Sokal D. Impact of Thailand's HIV-control program as indicated by the decline of sexually transmitted diseases. Lancet 1994;344:243—245. Loodts P, Van de Perre P. STD/HIV prevention, education and promotion of condom use among military recruits in Rwanda. Kigali, Rwanda: National AIDS Control Programme, Ministry of Defense. Reported in: Effective approaches to AIDS prevention. Geneva: World Health Organization/Global Programme on AIDS, 1992. Suggested Readings Hart G, Adler MW, et al. Evaluation of sexually transmitted diseases control programs in industrialized countries. In: Holmes KK, Mardh P-A, Sparling PF, et al. Sexually transmitted diseases. 2nd ed. New York: McGraw-Hill, 1990. Meheus A, Schulz KF, Cates W Jr. Development of prevention and control programs for sexually transmitted diseases in developing countries. In: Holmes KK, Mardh P-A, Sparling PF, et al.: 1990. Piot P, Hira S. Control and prevention of sexually transmitted diseases. In: Lamptey P, Piot P, eds. The handbook for AIDS Prevention in Africa. Research Triangle Park, North Carolina: Family Health International, 1990. Brathwaite AR, Piot P, Figueroa JP. Control and prevention of sexually transmitted diseases. In: Lamptey P, White F, Figueroa JP, Gringle R, eds. The handbook for AIDS Prevention in the Caribbean. Research Triangle Park, North Carolina: Family Health International, 1992. Heymann DL, Biritwum RB, Paget WJ. Evaluation of AIDS programs. In: Lamptey P, White F, Figueroa JP, Gringle R: 1992. Centers for Disease Control. Quality assurance guidelines for STD clinics. Atlanta, Georgia: Centers for Disease Control, 1986. Shekelle PG, Kosecoff J. Evaluating the treatment of sexually transmitted diseases at an urban public hospital outpatient clinic. Am J Public Health 1992;82:115—117. Bryce J, Toole MJ, Waldman RJ, Voigt A. Assessing the quality of facility-based child survival services. Health Policy Plan 1992; 2:155—163.

What's New

Also: