Impact evaluation

RCTs are studies used to measure the effectiveness of a new intervention. They are unlikely to prove causality on their own, however randomisation reduces bias while providing a tool for examining cause-effect relationships.[12] RCTs rely on random assignment, meaning that that evaluation almost always has to be designed ex ante, as it is rare that the natural assignment of a project would be on a random basis.[13] When designing an RCT, there are five key questions that need to be asked: What treatment is being tested, how many treatment arms will there be, what will be the unit of assignment, how large of a sample is needed, how will the test be randomised.[13] A well conducted RCT will yield a credible estimate regarding the average treatment effect within one specific population or unit of assignment.[14] A drawback of RCTs is 'the transportation problem', outlining that what works within one population does not necessarily work within another population, meaning that the average treatment effect is not applicable across differing units of assignment.[14]

Natural experiments are used because these methods relax the inherent tension uncontrolled field and controlled laboratory data collection approaches.[15] Natural experiments leverage events outside the researchers' and subjects' control to address several threats to internal validity, minimising the chance of confounding elements, while sacrificing a few of the features of field data, such as more natural ranges of treatment effects and the presence of organically formed context.[15] A main problem with natural experiments is the issue of replicability. Laboratory work, when properly described and repeated, should be able to produce similar results. Due to the uniqueness of natural experiments, replication is often limited to analysis of alternate data from a similar event.[15]

Quasi-experimental approaches can remove bias arising from selection on observables and, where panel data are available, time invariant unobservables. Quasi-experimental methods include matching, differencing, instrumental variables and the pipeline approach; they are usually carried out by multivariate regression analysis.

If selection characteristics are known and observed, they can be controlled for to remove the bias. Matching involves comparing program participants with non-participants based on observed selection characteristics. Propensity score matching (PSM) uses a statistical model to calculate the probability of participating on the basis of a set of observable characteristics and matches participants and non-participants with similar probability scores. Regression discontinuity design exploits a decision rule as to who does and does not get the intervention to compare outcomes for those just either side of this cut-off.

Difference in differences or double differences, which use data collected at baseline and end-line for intervention and comparison groups, can be used to account for selection bias under the assumption that unobservable factors determining selection are fixed over time (time invariant).

Instrumental variables estimation accounts for selection bias by modelling participation using factors ('instruments') that are correlated with selection but not the outcome, thus isolating the aspects of program participation which can be treated as exogenous.

The pipeline approach (stepped-wedge design) uses beneficiaries already chosen to participate in a project at a later stage as the comparison group. The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest. However, in practice, it cannot be guaranteed that treatment and comparison groups are comparable and some method of matching will need to be applied to verify comparability.

Non-experimental impact evaluations are so-called because they do not involve a comparison group that does not have access to the intervention. The method used in non-experimental evaluation is to compare intervention groups before and after implementation of the intervention. Intervention interrupted time-series (ITS) evaluations require multiple data points on treated individuals before and after the intervention, while before versus after (or pre-test post-test) designs simply require a single data point before and after. Post-test analyses include data after the intervention from the intervention group only. Non-experimental designs are the weakest evaluation design, because to show a causal relationship between intervention and outcomes convincingly, the evaluation must demonstrate that any likely alternate explanations for the outcomes are irrelevant. However, there remain applications to which this design is relevant, for example, in calculating time-savings from an intervention which improves access to amenities. In addition, there may be cases where non-experimental designs are the only feasible impact evaluation design, such as universally implemented programmes or national policy reforms in which no isolated comparison groups are likely to exist.

When there is an absence of the assumption of equivalence, the difference in outcome between the groups that would have occurred regardless creates a form of bias in the estimate of program effects. This is known as selection bias (Rossi et al., 2004). It creates a threat to the validity of the program effect estimate in any impact assessment using a non-equivalent group comparison design and appears in situations where some process responsible for influences that are not fully known selects which individuals will be in which group instead of the assignment to groups being determined by pure chance (Rossi et al., 2004). This may be because of participant self-selection, or it may be because of program placement (placement bias).[16]

Selection bias can occur through natural or deliberate processes that cause a loss of outcome data for members of the intervention and control groups that have already been formed. This is known as attrition and it can come about in two ways (Rossi et al., 2004): targets drop out of the intervention or control group cannot be reached or targets refuse to co-operate in outcome measurement. Differential attrition is assumed when attrition occurs as a result of something either than explicit chance process (Rossi et al., 2004). This means that "those individuals that were from the intervention group whose outcome data are missing cannot be assumed to have the same outcome-relevant characteristics as those from the control group whose outcome data are missing" (Rossi et al., 2004, p271). However, random assignment designs are not safe from selection bias which is induced by attrition (Rossi et al., 2004).

There are other factors that can be responsible for bias in the results of an impact assessment. These generally have to do with events or experiences other than receiving the program that occur during the intervention. These biases include secular trends, interfering events and maturation (Rossi et al., 2004).

Secular trends can be defined as being relatively long-term trends in the community, region or country. These are also termed secular drift and may produce changes that enhance or mask the apparent effects of a (Rossi et al., 2004). For example, when a community's birth rate is declining, a program to reduce fertility may appear effective because of bias stemming from that downward trend (Rossi et al., 2004, p273).

Interfering events are similar to secular trends; in this case it is the short-term events that can produce changes that may introduce bias into estimates of program effect, such as a power outage disrupting communications or hampering the delivery of food supplements may interfere with a nutrition program (Rossi et al., 2004, p273).

Impact evaluation needs to accommodate the fact that natural maturational and developmental processes can produce considerable change independently of the program. Including these changes in the estimates of program effects would result in bias estimates. An example of this form of bias would be a program to improve preventative health practices among adults may seem ineffective because health generally declines with age (Rossi et al., 2004, p273).

"Careful maintenance of comparable circumstances for program and control groups between random assignment and outcome measurement should prevent bias from the influence of other differential experiences or events on the groups. If either of these conditions is absent from the design, there is potential for bias in the estimates of program effect" (Rossi et al., 2004, p274).

The International Initiative for Impact Evaluation (3ie) defines rigorous impact evaluations as: "analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context".[20]

According to the World Bank's DIME Initiative, "Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program. Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental designs".[21]

Similarly, according to the US Environmental Protection Agency impact evaluation is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program.[22]

According to the World Bank's Independent Evaluation Group (IEG), impact evaluation is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project.[23]

Impact evaluation has been defined differently over the past few decades.[6] Other interpretations of impact evaluation include:

• An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, or a process evaluation which focuses on implementation;
• An evaluation carried out some time (five to ten years) after the intervention has been completed so as to allow time for impact to appear; and
• An evaluation considering all interventions within a given sector or geographical area.

Other authors make a distinction between "impact evaluation" and "impact assessment." "Impact evaluation" uses empirical techniques to estimate the effects of interventions and their statistical significance, whereas "impact assessment" includes a broader set of methods, including structural simulations and other approaches that cannot test for statistical significance.[16]

Common definitions of 'impact' used in evaluation generally refer to the totality of longer-term consequences associated with an intervention on quality-of-life outcomes. For example, the Organization for Economic Cooperation and Development's Development Assistance Committee (OECD-DAC) defines impact as the "positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended".[24] A number of international agencies have also adopted this definition of impact. For example, UNICEF defines impact as "The longer term results of a program – technical, economic, socio-cultural, institutional, environmental or other – whether intended or unintended. The intended impact should correspond to the program goal."[25] Similarly, Evaluationwiki.org defines impact evaluation as an evaluation that looks beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.[26]

Technically, an evaluation could be conducted to assess 'impact' as defined here without reference to a counterfactual. However, much of the existing literature (e.g. NONIE Guidelines on Impact Evaluation[27] adopts the OECD-DAC definition of impact while referring to the techniques used to attribute impact to an intervention as necessarily based on counterfactual analysis.

What is missing from the term 'impact' evaluation is the way 'impact' shows up long-term. For instance, most Monitoring and Evaluation 'logical framework' plans have inputs-outputs-outcomes and... impacts. While the first three appear during the project duration itself, impact takes far longer to take place. For instance, in a 5-year agricultural project, seeds are inputs, farmers trained in using them our outputs, changes in crop yields as a result of the seeds being planted properly in an outcome and families being more sustainably food secure over time is an impact. Such post-project impact evaluations are very rare. They are also called ex-post evaluations or we are coining the term sustained impact evaluations. While hundreds of thousands of documents call for them, rarely do donors have the funding flexibility - or interest - to return to see how sustained, and durable our interventions remained after project close out, after resources were withdrawn. There are many lessons to be learned for design, implementation, M&E and how to foster country-ownership.

There is intensive debate in academic circles around the appropriate methodologies for impact evaluation, between proponents of experimental methods on the one hand and proponents of more general methodologies on the other. William Easterly has referred to this as 'The Civil War in Development economics'. Proponents of experimental designs, sometimes referred to as 'randomistas',[8] argue randomization is the only means to ensure unobservable selection bias is accounted for, and that building up the flimsy experimental evidence base should be developed as a matter of priority.[28] In contrast, others argue that randomized assignment is seldom appropriate to development interventions and even when it is, experiments provide us with information on the results of a specific intervention applied to a specific context, and little of external relevance.[29] There has been criticism from evaluation bodies and others that some donors and academics over-emphasize favoured methods for impact evaluation,[30] and that this may in fact hinder learning and accountability.[31] In addition, there has been a debate around the appropriate role for qualitative methods within impact evaluations.[32][33]

While knowledge of effectiveness is vital, it is also important to understand the reasons for effectiveness and the circumstances under which results are likely to be replicated. In contrast with 'black box' impact evaluation approaches, which only report mean differences in outcomes between treatment and comparison groups, theory-based impact evaluation involves mapping out the causal chain from inputs to outcomes and impact and testing the underlying assumptions.[34][27] Most interventions within the realm of public policy are of a voluntary, rather than coercive (legally required) nature. In addition, interventions are often active rather than passive, requiring a greater rather than lesser degree of participation among beneficiaries and therefore behavior change as a pre-requisite for effectiveness. Public policy will therefore be successful to the extent that people are incentivized to change their behaviour favourably. A theory-based approach enables policy-makers to understand the reasons for differing levels of program participation (referred to as 'compliance' or 'adherence') and the processes determining behavior change. Theory-Based approaches use both quantitative and qualitative data collection, and the latter can be particularly useful in understanding the reasons for compliance and therefore whether and how the intervention may be replicated in other settings. Methods of qualitative data collection include focus groups, in-depth interviews, participatory rural appraisal (PRA) and field visits, as well as reading of anthropological and political literature.

White (2009b)[34] advocates more widespread application of a theory-based approach to impact evaluation as a means to improve policy relevance of impact evaluations, outlining six key principles of the theory-based approach:

1. Map out the causal chain (program theory) which explains how the intervention is expected to lead to the intended outcomes, and collect data to test the underlying assumptions of the causal links.
2. Understand context, including the social, political and economic setting of the intervention.
3. Anticipate heterogeneity to help in identifying sub-groups and adjusting the sample size to account for the levels of disaggregation to be used in the analysis.
4. Rigorous evaluation of impact using a credible counterfactual (as discussed above).
5. Rigorous factual analysis of links in the causal chain.
6. Use mixed methods (a combination of quantitative and qualitative methods).