�Creating a world free of HIV is one of the great moral callings of our time and one that requires a global response. On this World AIDS Day, I commend the people who are leading the fight in their nations and communities, and I especially call on the world community to rededicate efforts to prevent stigma and discrimination against people living with and affected by HIV/AIDS. The American people stand with the people of the world in the fight against HIV/AIDS.�
Secretary of State Condoleezza Rice
In order to ensure quality and sustainability of its programs, the President�s Emergency Plan for AIDS Relief (Emergency Plan/PEPFAR) is committed to the strategic collection and use of information for program accountability and improvement. With each year of implementation, PEPFAR expands its knowledge base of best practices and lessons learned, which drives funding decisions and adjustments to ongoing programs.
The collection and analysis of high-quality data in under-resourced settings is a critical challenge. For example, given the obstacles they face, health care personnel often find it difficult to track patients and manage records. In many countries, health management information systems (HMIS) are weak and thus compound the challenge.
To support host nations in addressing these issues, U.S. Government (USG) country teams, in coordination with host governments and partners, determine strategic information (SI) activities and priorities for each upcoming fiscal year as the part of the Country Operational Plan (COP) process.
PEPFAR-supported programs are monitored at different levels, providing information needed to improve programs.
This work starts with reporting core output results, in terms of three fundamental questions:
To monitor the outcomes and impact of PEPFAR and other international efforts at a higher level, the Emergency Plan supports serologic surveys to monitor trends in HIV prevalence within countries. Population-based surveys such as the Demographic and Health Survey (DHS) and AIDS Indicator Survey (AIS) measure changes in HIV-related behaviors. Program monitoring and evaluation systems also measure behavioral and health status changes of individuals who participate in PEPFAR-supported programs.
Many international partners are deeply involved in supporting the nations that have been hard hit by HIV/AIDS; this raises the specter of uncoordinated efforts, which can handicap international responses. One of the �Three Ones� principles for international coordination at the country level (discussed in the chapter on Strengthening Multilateral Action) is to support a single national monitoring and evaluation (M&E) system. Coordination with international partners is central to PEPFAR�s strategic information efforts.
Initiatives in 2006 to improve the information required for accountability and programming focused on:
Measuring Changes in the HIV Epidemic and Related Behaviors
Surveillance and Surveys
To address this challenge, PEPFAR uses Geographic Information Systems (GIS) as an information management and analytic tool to improve HIV/AIDS program delivery. By facilitating the integration and analysis of spatially referenced data by mapping, GIS help PEPFAR answer essential questions, such as: Where are the highest HIV-prevalence districts in the country? What is the geographic distribution of counseling and testing service delivery points? How many people live within the catchment areas of facilities that currently offer ARVs? Where should PEPFAR expand its services in order to maximize coverage or equity?
To obtain the kind of information needed for strategic decision- making, PEPFAR is using novel geo-spatial modeling techniques, in addition to techniques that have been applied to other diseases. GIS analytic tools improve the measurement and management of rapid service scale-up, which is essential as PEPFAR works to meet its prevention, treatment, and care targets. In the past, decisions regarding the distribution of services might have been based on an equity principle and/or convenience of access. Now, using the more quantitative, geographically linked GIS data, PEPFAR can target service delivery points to areas of greatest need.
For example, using GIS mapping, figure 7.1 communicates the gender disparity in HIV prevalence by geospatial dimensions; this data can be used to inform program planning. Figure 7.2 also demonstrates how GIS mapping can be used to track changes in treatment scale-up across all PEPFAR focus countries.
Starting in 2001, another source of data became available. The MEASURE DHS project, now funded by both the U.S. Agency for International Development (USAID) and PEPFAR, began including HIV testing of large, nationally representative samples of the population through the DHS. To date, 20 DHS and AIS surveys with HIV testing have been carried out in Latin America, sub-Saharan Africa, and Asia, and surveys are underway in another 11 countries. In almost all of these surveys, the percentage of women and men testing positive for HIV has been lower than levels estimated from sentinel surveillance. In Kenya, for example, national HIV prevalence estimated from sentinel surveillance ranged from 12 to 18 percent, as compared to the DHS population-based estimate of 6.7 percent.
What accounts for this difference? The most common sentinel surveillance sites, are predominantly urban and exclude men and non-pregnant women. Because HIV infection in developing countries is more common both among women and within urban areas, data collected from ANCs may lead monitors to overestimate national HIV prevalence. In contrast, the DHS tests a more representative sample of the population, including both women and men from all areas of the country.
All international health and scientific agencies, including UNAIDS, now agree that population-based testing provides a more accurate estimate of national HIV prevalence. In fact, based on the DHS results, UNAIDS has revised its national HIV prevalence estimates downwards for most countries.
In addition to providing a better estimate of overall HIV prevalence, the DHS also links HIV infection status with other social and behavioral information, such as education, knowledge, and condom use, giving a more detailed picture of the epidemic. This information is vital for developing appropriately-targeted programs across the spectrum of potential prevention, treatment, and care interventions.
To improve the quality of the data being collected to monitor HIV prevalence, morbidity, mortality, and behaviors, the USG developed training materials and participated in and supported regional surveillance trainings and workshops on estimating HIV prevalence and incidence. Training materials also are available on implementing basic HIV seroprevalance surveys, using Epi Info software to analyze ANC HIV sentinel surveillance data, sampling hard-to-reach populations, conducting behavioral surveillance, and performing sample vital registry with verbal autopsies. Additionally, the USG helps UNAIDS and the World Health Organization (WHO) to conduct regional workshops on HIV estimates and projections, and provides expert consultation on the modeling used to develop these estimates and projections. Finally, PEPFAR has supported the development of new surveillance methods, which include targeting hard-to-reach populations, monitoring antiretroviral drug (ARV) resistance, improving the quality of laboratory testing for HIV serologic surveys, and assessing new laboratory methods to monitor recent infections among HIV surveillance and survey samples.
Strengthening Country Strategic Information Systems
In 2006, the Emergency Plan worked with experts from host governments, implementing partners, and other multilateral and bilateral partners, to:
HMIS activities are supported at both the central level as well as within country programs. Examples of significant activities that have been undertaken at the central level include:
In-country Emergency Plan programs continue to support a host of innovative solutions that not only meet immediate program needs, but also build in-country capacity in health information systems. Some of these include the following efforts:
CareWare is free, client-level software for monitoring HIV clinical and supportive care. This program was developed under contract for the Health Resources and Services Administration within the Department of Health and Human Services, and was initially released in 2000 for use in U.S. HIV/AIDS clinics. Major clinical modules and reporting functions were added over time. In 2006, a major new version was released, configurable as standalone, on a wide area network, or on the Internet. A number of new features were added, included pharmacy prescription, invoicing, medication inventory, appointment scheduler, medication regimen builder, a Quality of Care module that tracks HIVQUAL indicators, a language translation module, and a forms designer to allow each local installation to customize the look of their data entry screens. CareWare is being implemented in Honduras, Kenya, Nigeria, Russia, Tanzania, Thailand, Uganda, Vietnam, and Zambia.
The Ethiopian Telehealth Project is a collaboration among the Emergency Plan, the U.S. Army Telemedicine and Advanced Technology Research Center, the National Defense Force of Ethiopia, the Ministry of Health, and Jimma University. Its goal is to use information and communications technologies to support the development of communities of practice within the country and to demonstrate the use of ICT to extend service delivery to peripheral settings, where most of the population lives.
The project has two basic components. The first is the creation of resource centers, from hospitals to health clinics. The resource centers, which vary in size from 30 seats in large teaching hospitals to a single computer at a rural health clinic, are connected via telecommunications links (mesh radio networks and national telecommunication backbone), and support the development of communities of practice through e-learning, simulation-based training, and tele-consultation. They also provide the ability to effectively exchange patient record data.
The second component of the project is the deployment of mobile HIV/ART clinics. These clinics are based at health clinics and will allow clinic nurses to provide HIV/AIDS services, equivalent to those provided in hospitals, on a rotating basis at community-level health posts. In Ethiopia, where nearly 50 percent of the people have no direct access to a health clinic, the ability to offer consistent health care services and the capacity to link directly to higher-level facilities will greatly increase and improve HIV/AIDS management.
The Ethiopian Telehealth Project will demonstrate that:
The portable HIV/AIDS clinic project will demonstrate that:
Improving Data Quality of PEPFAR Results Reporting
The setting of targets is complex, and the quality of partner results or output data varies. Because the United States remains the only international HIV partner to require that its partners report annually on standardized indicators, the introduction of data quality standards is an ongoing effort. However, one benefit of standardized indicators is that PEPFAR tracks key services by age and by gender.
PEPFAR has been transparent regarding the challenges it has faced in regard to data quality, especially in the areas of prevention and care. The only OIG audit reports to look at treatment were Ethiopia and Haiti, and neither had reporting issues. The Emergency Plan has taken steps to correct obvious data deficiencies.
For instance, in C�te d�Ivoire, national numbers are not included in upstream totals because they are not reliable. Other national totals also have been reduced to offset the duplication of counts. Even before the OIG audits, PEPFAR had already recognized these challenges and taken steps in improving and monitoring community-level programs. Strategic information advisors were sent to help countries strategize about how to strengthen their program measurements.
In 2006, for example, South Africa added a Data-Quality Advisor position to its PEPFAR USG team and constructed a single data warehouse into which all PEPFAR partners report on their performance, thereby ensuring an audit trail on all data reported by the Mission. As the OIG noted, there are as many cases of under-reporting numbers as there are of over-reporting by partners.
All of the 15 focus countries, along with 16 additional countries receiving more than $5 million annually in USG bilateral HIV/AIDS funding, now have a headquarters advisor to assist them with in-country partner trainings on data collection and data quality. The Strategic Information advisor works on record-keeping, program reporting standards, HMIS systems, and other quality improvement initiatives.
PEPFAR continues to improve and clarify target and reporting dates and associated guidance. The number of targets and reporting measures has been reduced, from the original set of 63 measures in 2004 to the current set of 41 measures.
Each year, the guidelines for measuring progress are improved and re-issued. This guidance builds upon a reporting innovation introduced by PEPFAR, which clarifies both target-setting and related results: the use of the terms upstream (indirect) and downstream (direct) to distinguish between strengthening the service delivery system (e.g., laboratory support for HIV testing) and providing direct services at the site of delivery.
Addressing data quality challenges, in 2006 PEPFAR introduced tools for ensuring data quality, such as:
These tools were co-sponsored by PEPFAR, the Global Fund, and the Health Metrics Network.
In addition, the National M&E Systems Assessment Tool, which assists in the evaluation of M&E plans and systems (figure 7.3) and has been endorsed by the Emergency Plan, the Global Fund, UNAIDS, WHO, the World Bank, Health Metrics Network, and Roll Back Malaria, was published in December 2006. This tool was pilot-tested with the help of Global Fund and USG partners in eight countries: Bangladesh, Chile, China, Democratic Republic of Congo, Niger, Russia, and Rwanda. The tool was designed to assess data collection, reporting, and management systems to measure indicators of program/project success. This tool addresses primarily the M&E plan and systems that need to be in place to collect and report data for aggregation into indicators.
The objectives of the M&E Systems Strengthening Tool are to:
The second tool, the Data Quality Audit Tool, was successfully- pilot tested in November 2006 in Tanzania and is being finalized. It provides an audit form and guidance for USG audits of their partners and was developed by MEASURE consultants, including a former USG program auditor.
The third tool, the Data Quality Assurance Tool for Program-Level Indicators, is a PEPFAR-specific tool that outlines essential parameters of data quality and how data quality fits within the Emergency Plan system of results reporting. This document provides USG country teams with the tools they need to improve the data quality of PEPFAR results reporting.
From June to August 2005, assessments of existing USG and national data collection and reporting systems were conducted in Botswana, Kenya, South Africa, and Zambia.
Following these visits the team developed the Data Quality Assessment Tool, which was then reviewed by additional USG country teams, edited, and finalized in spring 2006. The Data Quality Assurance Tool addresses three important issues in assessing and improving the quality of data:
These products (figure 7.4) will help PEPFAR to ensure that systems and processes contribute to long-term, sustainable, high-quality HIV/AIDS monitoring and evaluation capacity in host nations. Together, they provide a comprehensive approach to improving PEPFAR data quality.
Data Quality Training
The trainings, which were attended by several hundred field staff, addressed indicators and reporting to PEPFAR, data quality assessments, M&E capacity-building, agency and national coordination, and target setting. A series of follow-up distance-based trainings have followed. PEPFAR now holds monthly digital video conferences and/or conference calls with headquarters and country staff, addressing issues related to both challenges and best practices, regarding implementation, management, and strategic information (SI). These sessions give staff regular opportunities to present and exchange information.
Approximately 27,200 people from the focus countries have been trained and/or retrained as a result of 2004 and 2005 SI training. The number of people participating in SI training in 2006 continued to increase.
Expanded Use of PEPFAR Results Reporting
Additionally, because all USG programs providing global HIV/AIDS assistance fall under the auspices of the Emergency Plan, the COPRS planning and reporting model (figure 7.5) was expanded in 2006 to cover five additional countries receiving more than $10 million annually � Cambodia, India, Malawi, Russia, and Zimbabwe. These countries completed 2006 mini-COPs, a reduced version of the full COP. In addition, six countries that received more than $5 million in HIV/AIDS funding annually completed five-year planning strategies � China, the Democratic Republic of Congo, the Dominican Republic, Lesotho, Senegal, and Swaziland. For fiscal year 2007, a total of 16 countries are completing mini-COPs. Development of a strategy and/or an annual COP provides an important opportunity to bring the USG country team and key host country and international partners together. This joint planning effort highlights areas for USG investments and support within the context of one HIV planning and coordinated country system.
Just as the number of countries involved in joint COP planning has expanded, the number of USG-presence countries reporting results has expanded. In 2006, USG teams in the 42 countries receiving over $1 million in HIV funding annually reported HIV results, using a common set of indicators that measure prevention, treatment, and care activities. Data quality guidance and related training have been undertaken for these countries.
The Public Health Evaluation Strategy
The objective of this transformation is to further strengthen the Emergency Plan�s strategy to support the most scientifically sound and cost-effective methodologies as the epidemic, the responses, and new opportunities develop. Of note, the Public Health Evaluation approach will not support basic or investigational clinical research activities, which are already well-supported by the National Institutes of Health within the U.S. Department of Health and Human Services.
The shift to Public Health Evaluation includes a commitment to implement a routine system to allow for identification of strategically important questions, coordination of efforts (across projects, partners, and countries) to answer priority questions, and dissemination and application of methods, tools, and findings. Key elements of the system include a multi-agency Public Health Evaluation Subcommittee, which operates under the authority of the Scientific Steering Committee, and Evaluation Teams comprised of technical experts, field staff, and partners to oversee specific questions/topical studies. The Public Health Evaluation Subcommittee was formed in late 2006, and quickly reviewed more than 200 country and centrally submitted studies, identified areas of multi-country focus, and assessed gaps in the current evaluation portfolio. The first Evaluation Teams are expected to be set up in early 2007. An annual priority-setting process will be undertaken in order to solicit priority questions from agencies, field staff, partners, and other experts.
Communicating Results and Best Practices
One new initiative announced at the Meeting was the PEPFAR Extranet. This website is an online community where USG personnel can exchange information and best practices about HIV programs. Country staff may list best practices and local presentations, and use the site for online discussions. Technical working groups also can post best practices and results. For example, information on country electronic information systems can be found on the PEPFAR Extranet.
The Implementers� Meeting was followed by regional targeted trainings in Latin America and Caribbean, East/West Africa, Eastern Europe/Eurasia and Asia. These sessions provided information about PEPFAR policies, program planning, and reporting requirements for countries receiving more than $1 million in annual funding through PEPFAR. Technical staff from the 15 focus countries, along with headquarters advisors, acted as trainers, enabling peer-to- peer exchanges of best practices. The Asia training was jointly sponsored and conducted with UNAIDS. Participants commented that the opportunity to share information with their peers in other countries through networking, presentations, and group exercises was particularly helpful. Also helpful to participants, most of whom were new to the PEPFAR concept of �One USG Team,� were the discussions of PEPFAR�s organizational structure and evolving policy issues. These meetings paved the way for the fiscal year 2007 expansion of the number of countries that complete COPs and report annual program and outcome results.
In 2006, the Implementers� Meeting and the regional trainings were supplemented with postings of abstracted results from intervention research by the Cochran Group � University of California San Francisco and distance-based trainings. COP digital video trainings reached USG staff located throughout the world. Twice monthly, literature abstracts highlighted new HIV intervention findings, to keep staff in the field current. Annual meetings with international partners on the joint analysis of treatment and care results continued, as did other joint international efforts to strengthen country SI systems. The USG also continued collaborating with the Global Fund, UNAIDS, UNICEF, the World Bank, and WHO to produce coordinated guidelines for reporting future results.
Key Challenges and Future Directions
Disruptions to national health systems have included major setbacks to efforts to monitor and evaluate programs, since M&E activities are often the first programs to be abandoned during emergencies. For example, during fiscal year 2006, C�te d�Ivoire experienced difficult challenges due to civil unrest, which complicated the reporting task of in-country teams.
| U.S. Government interagency website managed by the Office of U.S. Global AIDS Coordinator|
and the Bureau of Public Affairs, U.S. State Department.