Performance evaluations are a key tool in our repertoire of evaluation designs when assessing program implementation and performance. We use it when answering descriptive and normative questions. What has the program achieved? How is it being implemented? How is it perceived and valued by beneficiaries and stakeholders? We also use performance evaluations to identify the contribution of a program to results when an impact evaluation is not an option.
When designing our evaluations, we use a collaborative, stakeholder-centered approach, integrating their feedback into our design, execution and analysis. In this way, we ensure that the evaluation meets stakeholder needs and generates results that can be used to inform future program design and decision-making.
Abt believes that performance evaluations must be held to the highest standards of rigor. Our performance evaluation designs include interruptive time-series designs, before-and-after designs, cross-sectional designs, snapshots and case-study designs. In conducting these evaluations, we draw on our deep bench of methodological talent who help our clients employ the most rigorous design feasible using a complementary mix of quantitative and qualitative methods, strategies to utilize secondary data, and advanced data-driven technology solutions.
Poppy production in Afghanistan has reached record highs, poverty has risen in rural areas in recent years, and farmers continue to face food insecurity. The USAID-funded Regional Agricultural Development Program offers farmers alternatives to poppy production by promoting crop diversification and strengthening value chains connecting farmers, processors and wholesalers, to help increase rural incomes and employment across 21 provinces and four regions in Afghanistan. Since an impact evaluation was not feasible, Abt designed a cross-sectional performance evaluation, combining a rigorous qualitative approach to assess outcomes with GIS and remote sensing analysis to measure changes in land area used for poppy production.
Using a purposive sample, we conducted interviews and focus group discussions with hundreds of farmers, agribusiness and key-stakeholders. This sample gave us a nuanced picture of how the program was perceived and valued by beneficiaries, if the program was achieving its targets, and if the approach and implementation was effective in achieving program goals. The GIS and remote sensing analysis compared the change in the area under poppy production before and during the program‘s implementation, and compared changes across areas that were part of the program to those that were not. The evaluation findings provided rich data on the geographic variations in the program’s progress and accomplishments and provided recommendations to scale-up or modify activities.
Etosha National Park, one of Namibia’s primary attractions, covers 8,600 square miles of wilderness, including the massive Etosha salt pan, savanna woodlands and water holes where tourists watch animals such as rhinos, elephants, and big cats. Despite these attractions, there is fierce competition in the African safari business, pitting Namibia against better-known parks in neighboring Botswana and South Africa. Abt is evaluating the Millennium Challenge Corporation’s (MCC) five-year initiative which aimed to grow the tourism sector in Namibia through park improvements and a marketing campaign.
Our ex-post performance evaluation uses a mixed-methods approach combining qualitative data from an array of stakeholders and beneficiaries, online surveys and site observations to assess MCC’s investment in Etosha National Park and its marketing campaign to promote Namibia as a tourist destination. We are also examining trends in tourist arrivals and park visitors and revenue using secondary data from the Namibian government. The evaluation will show MCC and Namibian stakeholders what worked well and what did not promote the growth of the tourism sector, and will inform future efforts to tackle challenges at the intersection of tourism and economic growth.
Diarrheal diseases are a leading cause of child mortality in Ghana among children under 5 years of age. As part of USAID’s Strengthening Health Outcomes through the Private Sector (SHOPS) project, Abt evaluated a comprehensive program in Ghana to increase the use of oral dehydration salts (ORS) with zinc and new diarrhea treatment guidelines to the private sector, to complement public sector efforts. We also worked with local pharmaceutical manufacturers to introduce zinc in the market, trained private health care providers on diarrhea practices, and developed and aired a national mass media campaign to generate demand for zinc and promote its use with ORS.
Abt used a before-after design to examine how demand for and use of ORS and zinc for treatment of diarrhea among caregivers changed over the course of the project. Results from large-sample household surveys among caregivers showed substantial increases in the use of ORS and zinc. Validated in part by our findings, USAID is now supporting the scale-up of this approach in Ghana.
In Western Papa New Guinea, the capacity of primary health service providers is inadequate and access and utilization of health care services is poor. The North Fly and Middle and South Fly Health programs responded to the need to address these lacunae in health services.
Abt conducted a base-, mid- and end-line evaluation to assess the challenges and successes of program implementation and the extent to which the program contributed to strengthening primary health care outcomes. We conducted a mixed-methods cross-sectional evaluation using a range of qualitative data collection tools—including key informant interviews and focus group discussions—and incorporated critical national health data and all-years monitoring data sources.
Combining these data with facility assessments and community level surveys that reached hundreds of community members, we were able to provide an in-depth assessment of changes in perceptions of access, quality and utilization of health care services. We also analyzed clinical program data and district-level and monthly facility-level data to triangulate our overall findings that the program contributed significantly to direct health services in under-supported areas.
Select from our growing list of insights below.
Abt Associates is an engine for social impact, harnessing the power of data and grounded insight to bring people from vulnerability to security worldwide. We think boldly to deliver solutions that cross disciplines, methods, and geographies. Abt provides impact through research, evaluation, and program implementation in the fields of health, social and environmental policy, and international development.