FHI/Vietnam and FHI/Nepal Collaborate on Data Quality Audit

JULY 2008 — When decisionmakers, implementers, and funders need data from an FHI country program, they reap what has been sown. Rather than GI/GO (garbage in, garbage out), FHI prefers the QI/QO (quality in, quality out) model. For this reason, FHI invests much effort into ensuring the data it gathers is valid, reliable, accurate, precise, and timely. If the data falls short, programs may falter and funders may start looking for the exit.

Thankfully, FHI program managers, field supervisors, and other staff have a sharp tool—the "data quality audit"—to maintain and improve the quality of program data year after year, keeping programs strong and providing sound quantitative evidence on global progress being made against HIV and AIDS and other diseases.

While the topic of data quality may not seem like much fun, the recent collaboration between FHI/Nepal and FHI/Vietnam on a data quality audit shows that such "south-to-south" exchanges can lead to very lively, collaborative, and fruitful exchanges.

Many Faces at the Table
FHI/Cambodia and FHI/India offered support to FHI/Nepal in September 2007 when it conducted a data quality audit on its ASHA (Advancing Surveillance, Policies, Prevention, Care and Support to Fight HIV/AIDS) program. In turn, FHI/Nepal shared its accumulated experience with FHI/Vietnam when a program data audit was needed there later in the year. With an FHI/Nepal colleague providing technical assistance, a host of Vietnam implementing partners, health educators, and FHI program officers and monitoring and evaluation staff collaborated on a highly participatory and supportive data quality audit from the start.

How They Did It
The audit included three steps: interviewing management and program staff, reviewing data records, and observing data management practices, including hard copy and electronic files. They took a close look, for example, at monthly project indicator forms, financial ledgers, client record files, registers from drop-in centers, and the daily diaries of outreach workers.

The auditors were checking that the right variables were measured, ensuring that data were exempt from bias on the part of the interviewer or recorder, and looking for transcription or sampling mistakes as well as for signs of manipulation for political or personal reasons. These steps verified the data's validity, accuracy, and integrity. They also looked for consistency in collection and measurement procedures, which give a measure of the data's reliability. They checked completeness by observing whether results came from a complete list of eligible persons or just a fraction of that list. They looked at the level of detail in the data, to get a measure of its precision; for example, they made sure the sex of individuals receiving counseling and testing was duly recorded. To get a sense of the system's timeliness, they asked questions about the rate of information system updates, the change of program activities, and how much information was actually being used or required.

How They Prepared
Not surprisingly, partner staff members reported that going into the audits, they were anxious about what the questions would be, the presence of outside auditors in their workplace, and the investigation methods that would be used. Despite their concerns, however, they hoped some important lessons would come out of the experience.

Before site visits took place, the auditors conducted extensive groundwork. The team took an in-depth look at the overall program: its technical elements, types of partners, reporting methods, and information systems. They took the time to become familiar with their local partners' monitoring and evaluation routines, indicators, and information systems.

To verify the quality of data, they decided to track five key data quality indicators. Four were part of PEPFAR's reporting requirement (number of female sex workers, men who have sex with men, injecting drug users reached through community outreach promoting HIV/AIDS prevention, and number of individuals receiving HIV test results with post-test counseling). The fifth was number of visits at drop-in centers. Data quality was verified for the most recent quarter and for the last month.

The Joy of Collaboration
The Vietnam data quality audit proceeded smoothly and on time. It shed light on some best practices. For example, a simple and easy "recall method" (wherein clients, not peer outreach workers, supply estimates of intervention coverage, thereby minimizing double-counting) was used throughout the country, and a standard voluntary counseling and testing services database had been developed with the Centers for Disease Control and Prevention.

The audit also clarified a number of opportunities for improvement. Chief among them were the need to establish a country-wide management information system similar to the one in use in Nepal, and to gather more narrative details in support of quantitative results.

The audit group's success was greater than the sum of its recommendations, however. There was unprecedented teamwork between program officers, monitoring and evaluation staff, and FHI country directors in Nepal and other countries in the region. This experience strengthened staff technical skills, improved morale, and developed local capacity to that can be brought to bear on future data audits in the region.

A High-Value Exercise for the Country Offices
Receiving technical assistance from a neighboring FHI country office was an innovation. FHI/Nepal's proximity meant its expertise was cost effective and accessible. The staff's familiarity with the sociology, language, and culture of Vietnam made conversations easy and ideas fun to pursue. The exchange of experience between monitoring and evaluation officers, in particular, sharpened the exercise from start to finish.

At the end of the day, there were important gains for country program staff. FHI community partners understood the need for high-quality data, accepted the importance of data quality audits, and overcame their initial concerns because of the positive experience. In the end, they said the audit was "supportive, positive, and empowering." They had come to see that data quality is a collective and a corporate responsibility that improves the quality of data, decisionmaking, and planning.

Bharat Raj Gautam, FHI's Nepal-based senior monitoring and evaluation officer, played a pivotal role in the Vietnamese audit, and is justifiably proud of its success. He says, "The guidance, instructions, suggestions, and motivation that came 'vertically' from FHI regional and headquarters offices 'fueled' the successful audit. But the energy to ignite the fire required 'horizontal' bonds between country offices—of shared experience, exposure to different types of interventions, and stimulating collaboration."

PHOTOS: Scenes from the FHI/Vietnam internal data quality audit, 2007. (FHI/Vietnam)