Summary Recommendations of the Data Working Group (DWG)
Summary Recommendations of the Data Working Group (DWG)
As part of PEPFAR’s ongoing effort to increase the efficiency and impact of USG investments in the global fight against AIDS, the data working group (DWG) of PEPFAR’s Scientific Advisory Board (SAB) offers four recommendations to Ambassador Eric Goosby. The DWG’s recommendations are derived from the following five principles of data collection and management:
• Data should strengthen US government program management;
• Data are a public good;
• Transparency and accountability;
• Standardization and comparability through a common data platform; and
The DWG’s four recommendations (elaborated below) are:
1. Strengthen and streamline HIV/AIDS program indicators;
2. Expand the collection, application, and sharing of budget, expenditure and cost data through the Expenditure Analysis initiative;
3. Establish and maintain a PEPFAR public access knowledge portal; and
4. Require each future grantee and contractor to submit a standardized, realistic, and contractually binding “Data Management Plan” that conforms to specific PEPFAR guidance.
1. Strengthen and streamline HIV/AIDS program indicators
In its 2009 Next Generation Indicators Reference Guide, PEPFAR defined ~35 “essential” and ~30 “recommended” (direct) program indicators. These indicators have been compiled from several sources, including the WHO, UNGASS core national indicators, PEPFAR technical working groups, and UNAIDS. In addition, implementing agencies (i.e. USAID and the CDC) and country teams may ask grantees to report against other indicators. In total, there are anecdotal reports that some contractors must report on up to 300 indicators for a single grant. This data collection and oversight is quite costly for all parties, including implementing partners, country teams, the implementing agencies, and PEPFAR’s strategic information system. The high cost is particularly problematic given the lack of a cohesive strategy for data use – that is, who should be monitoring and analyzing what pieces of data, at what level, and for what purpose. The quality of data collection is also unclear; it is likely that data quality declines as the number of indicators increases (or, alternatively, that greater demands for data require trade-offs in terms of program implementation). For example, an examination of central PEPFAR data on ART retention (arguably one of the most important indicators) for FY2011 shows several inconsistencies – including no data from Kenya, clearly incomplete reporting from Mozambique, and an unexpectedly large denominator in Namibia (180% of FY2010 reported ART initiation).
The DWG recommends that OGAC commission an external study of the HIV/AIDS indicator system. The study should begin by mapping an ideal data use strategy, and subsequently recommending a minimum set of indicators to serve the specified data needs. The study should thoroughly examine the indicator system from the bottom up, with particular emphasis on 1) data collection and utilization for performance management; 2) country level reporting and aggregation systems; 3) minimization of parallel reporting systems and duplication; and 4) integration of program and epidemiological indicators with expenditure data. Such a study could be conducted by the Institute of Medicine (IOM), 3ie or another analytical group. The study should rank indicators (and their various disaggregations) based on the needs assessment paired with a cost-benefit analysis, considering the following indicative criteria:
o Alignment with Global Fund, UNAIDS, and country data systems
o Importance for real-time program management or epidemiological understanding
o Accuracy of complete reports (based on audit?)
o Clear and readily understandable definition
o Proximity to being an “outcome” on the Input à Output à Outcome scale
o Degree of match to an activity to which expenditure data is being or could be assigned/collected
o Whether indicator is already part of the national health information system in a given country
o The marginal cost of collecting the indicator (i.e. does this indicator require a “special” data collection system, or could it be collected using the same method as other important indicators?)
o Redundancy, i.e. whether the information contained in this indicator could be largely inferred from the value of another indicator
PEPFAR should use the findings of this external assessment to streamline both its indicator list and data collection and reporting process. OGAC should also work closely with its country teams and implementing agencies to ensure interagency coordination, particularly with respect to limiting the number of extraneous indicators required or requested by non-OGAC parties.
2. Expand the collection, application, and sharing of budget, expenditure and cost data through the Expenditure Analysis initiative
In recent years, PEPFAR has made substantial investments in understanding the cost structure of its programs and improving their cost-effectiveness. The most ambitious effort has been the Expenditure Analysis (EA) initiative, which released its pilot report from the first six countries in July 2012. PEPFAR is currently scaling EA to more countries, with the ultimate goal of routinizing EA into all PEPFAR programs. The DWG applauds OGAC for its efforts to collect more financial data and seriously address value for money in US AIDS investment.
The DWG recommends that PEPFAR continue to expand the collection, application, and sharing of EA data. EA scale-up should gradually include more countries, more programs, greater detail for cost components, and more service delivery areas. This data should then be systematically analyzed and applied to improve the value for money of PEPFAR programs. Such USG use of EA data can drive value for money gains in three key ways: 1) by identifying inefficient programs at the country level and modifying or eliminating expensive cost outliers; 2) by revealing PEPFAR’s cost structure to inform resource allocation decisions and technical guidance; and 3) by shedding light on the drivers of cost-effectiveness at the program level, which can help improve future program design.
To achieve these value for money gains, PEPFAR should provide guidance to country offices on how to use EA results as a tool for program management. Many valid reasons exist for variation in cost between implementing partners, such as urban vs. rural catchment populations or cost-sharing with other partners (i.e. receiving ARV through a Global Fund grant). Nonetheless, country offices with knowledge of local conditions should be empowered to undertake program management actions informed by EA data, including reprogramming or termination of inefficient programs when appropriate. Country officers can then reallocate those funds to partners shown to be both effective and efficient. At the central level, OGAC should analyze the portfolio-wide results of EA to assess the relative cost-effectiveness of different interventions and delivery strategies. Such findings should be incorporated into COP guidance to inform program planning and resource allocation, both across and within partner countries. OGAC should carefully track and quantitate the impact of EA on the technical efficiency of PEPFAR programs.
The DWG also urges PEPFAR to share EA data with partners and stakeholders. In particular, country governments and the Global Fund have much to gain from access to EA data. As PEPFAR works to increase host country financing and ownership of AIDS programs, country governments must understand the financial implications and cost structure of program transitions. Likewise, the Global Fund could use EA data to help benchmark “reasonable” unit costs for grant outputs in different country contexts, allowing the Fund to make well-informed financing decisions. Finally, because of the broad public interest in understanding the costs of AIDS programs, the DWG recommends that PEPFAR publicly disclose the properly anonymized underlying data for analysis and use by external experts.
3. Establish and maintain a PEPFAR public access knowledge portal
PEPFAR’s current website is an invaluable resource for those interested in the USG contribution to the fight against AIDS. Among other resources, the website provides program policies, high-level budgets and expenditure, a range of fact sheets, a partial list of relevant research papers, and short analyses of findings from undisclosed costing data.
However, public accessibility for program data on the PEPFAR web portal does not compare favorably to those of its partners and foreign counterparts. In particular, DFID, the Global Fund, World Bank, and US Millennium Challenge Corporation (MCC) have been pioneers in open government; their respective websites offer detailed, interactive data on all funded programs in a user-friendly format. In contrast, most PEPFAR data is hidden in lengthy, complicated PDFs. While this information is technically public, it is often difficult and frustrating to find, extract, and interpret, particularly for the citizen layperson. This is particularly problematic because the scope and detail of PEPFAR’s formal annual reporting has declined over time, from 216 pages and 82 indicators in 2007 to 5 pages and 7 indicators in 2012.
Based on two of the guiding principles outlined above – 1) that data are a public good; and 2) that data should strengthen transparency and public accountability – PEPFAR should establish, beta-test, and then expand and maintain a public access portal which gradually expands the scope and granularity of disclosed PEPFAR program indicators; provides a graphical interface allowing the user to browse the data; and provides a “wiki” platform to which PEPFAR grantees, employees, partners, etc. would be encouraged to contribute data, “grey literature” reports and research papers at a more granular level. The precise level of detail and transparency would be subject to internal PEPFAR decision-making. A relatively uncontroversial first step could be to provide the information already available in PDF through this more user-friendly format. The DWG also recommends that properly anonymized data from PEFPAR-funded research (i.e. combination prevention trials) be made publicly available through the public access knowledge portal.
4. Require each future grantee and contractor to submit a standardized, realistic, and contractually binding “Data Management Plan” that conforms to specific PEPFAR guidance
Already, most contractors are required to submit a “performance monitoring plan” or similar document as part of the grant proposal process. Typically, these documents outline a definition of each performance indicator; the source, method, frequency and schedule of data collection; and targets for each performance indicator. For our purposes, we refer here to all such documents as “data management plans,” or DMPs.
DMPs are a powerful tool for program management and routine data collection. However, in large part because PEPFAR does not appear to provide centralized guidance for the preparation of such documents, the current system has several deficiencies. First, grantees face incentives to develop unrealistic DMPs during the proposal process, as “more” indicators can be perceived as a “stronger” grant proposal, particularly if the DMP is not closely analyzed for technical feasibility during the proposal review process. This is exacerbated by the wide discretion of country teams and agency contract officers, who may also request the inclusion of a wide range of indicators and disaggregations. Data quality will inevitably fall if M&E efforts are spread over such a broad range of indicators. Second, in part due to their unmanageable scope, DMPs are not currently treated (in practice) as contractually binding in either direction. It is not clear whether USG country officers have clear recourse against incomplete or low quality data. In the other direction, USG country teams often request further data that is outside the scope of the formal DMPs, which may place an unreasonable burden on implementing partners. Finally, DMPs rarely extend beyond “indicators” to sufficiently account for the collection and archiving of patient or facility-level data, such as on patient retention and outcomes. While this level of granularity is unnecessary for routine reporting toward PEPFAR’s aggregation of essential indicators, such data can be of enormous value to researchers, both within PEPFAR and among the broader scientific community.
Following the comprehensive assessment of the indicator system proposed in Recommendation 1, the DWG recommends that PEFPAR produce clear, centralized guidance for DMPs, including the circumstances under which deviations or exemptions from the DMP would be allowed. The guidance should focus DMPs around high-quality data collection and reporting for PEPFAR’s essential indicators. Implementing partners should be expected to include all applicable essential indicators in their DMPs. Additional indicators can be included, but should require justification for their necessity either by the implementing partner or USG country team. By limiting the number of indicators to those PEPFAR has itself defined as essential (in accordance with Recommendation 1), PEPFAR can reasonably demand high-quality data on a small selection of the most important indicators. PEPFAR should treat the DMPs as contractually binding on both ends; implementing partners should be penalized for failing to comply with the terms of the DMP, and USG country officers should refrain from requesting data outside the scope of the DMP. Ideally, DMPs should also facilitate data reporting in a standardized form, which can gradually feed into PEPFAR’s strategic information system (FACTS INFO) and the public access knowledge portal (Recommendation 3).
In addition to contractually binding indicator reporting, DMPs should provide a detailed description of implementers’ internal collection and management of patient-level data for their own purposes, including the nature and scope of data collected; how and why data will be shared and archived; metadata standards; intellectual property rights; data format; and data security, ethics, and privacy. Where possible, properly anonymized patient-level data should be electronically archived (and ideally shared via the public access knowledge portal. If an implementer is unable to ensure electronic archiving, the DMP should clearly state the rationale (i.e. prohibitive cost, concerns over anonymity, or national legislation which prohibits public archiving). PEPFAR can then use the completed DMPs to maintain a central list of potential data sources for PEPFAR-funded research projects and the broader scientific community.
 The findings and recommendations in this presentation are a compilation of views discussed in the working group and endorsed by most participants; they are neither the unanimous consensus of the DWG participants nor official positions of the US government. SAB participants in the DWG were Ward Cates, Kevin DeCock (also USG), Mead Over, Mary Jane Rotheram, Sten Vermund, Brian Williams, and Robin Wood. US government participants in the DWG were Paul Bouey and Rob Lyerla.
 The Institute of Medicine recently conducted a similar exercise for domestic HIV programs, resulting in a focused list of indicators and recommendations for streamlining M&E processes. The domestic report may be instructive for PEPFAR both in its processes and in the resultant list of indicators. See Institute of Medicine (2012). Monitoring HIV Care in the United States: Indicators and Data Systems. Washington: National Academy of Sciences.