Negombo clinic resources for participants

Participants of the recent communications for policy influence clinic in Negombo can find the full set of presentations, handouts and resources here. Videos are also now available.

more...

Video footage from Rome clinic now available

Interviews with participants of the communications for policy influence clinic in Rome can be viewed here. You will also find all presentations, handouts and resources from the clinic.

more...

2. Risks

Guidelines

Right up front, you will need to assess those external factors that could affect your ability to disseminate, access and influence policy makers to incorporate your evidence in policy decisions.  In any independent evaluation, there is a risk that the evaluation will find that the project had zero impact, or even a negative impact. There is also a risk that the project will be scaled up even in the face of such a finding. This section should asses the expected external risks associated with the policy engagement and dissemination of the findings. These external risks could depend on the limited freedom of press, an upcoming election, high level of political turnover, political violence or others.

The section should also identify a mitigation strategy to ensure that the findings of the evaluations are disseminated as widely as possible. Risks regarding the implementation of the evaluation should not be included here.

Example

Evaluating the Effectiveness of a Community-Based Conditional Cash Transfer Program in Tanzania

In this PIP the grantee outlines the mitigation strategies for dealing with risks related to contamination of control villages, and lack of effective randomization or take up of study findings due to political interference.

One of the primary risks we have been careful to guard against during the course of the intervention is contamination. First, what if the implementing agency was to roll out this or another CCT in control villages? To avoid this problem, we have met repeatedly with representatives of the Government of Tanzania (GoT), and they have approved our evaluation plan. The midline survey – already carried out – has not indicated any problem of targeted social programs in control villages. Since both the baseline and midline surveys have already been carried out, and the endline survey is scheduled to begin within the next four months, the risks of any contamination of the sample are largely mitigated.

A second potential risk to the relevancy of the evaluation would be if the control and treatment groups were not effectively randomized, potentially due to interference from other interested parties. However, randomization has already occurred and this does not seem to have been a problem. Detailed household data confirm balance (across treatment and control villages) on important variables. This broad balance across baseline covariates suggests that randomization was successful, and allows attribution of impacts to the intervention and not political interference in the sample. Further, to eliminate any small, pre-existing differences, we will use difference-in-difference analysis to confirm that the results are not sensitive to any imperfections of the randomization. A third risk is that the implementing agency could discontinue the program due to a lack of resources. This concern is allayed by the fact that the transfers for the course of the evaluation were earmarked from the outset, by the GoT’s recent decision to scale up the program to the national level, and by the fact that the program has been underway for two years with no such problem. The GoT has sufficient resources to keep the program going in our program villages, and even a desire to increase the amount of resources dedicated to it. Further, the implementing agency has been in continual contact with us about the implementation, and has been incredibly responsive in all communications.

Another potential risk is that the findings do not align well with political interests, particularly if we find little impact of the intervention, or if there is a change in the government. In the case that the impact evaluation does not find a significant impact of CCTs on health outcomes and educational attainment, the government might not be as keen to report and widely disseminate the results as they would in the case of positive results. To mitigate this risk, our impact evaluation strategy includes comparing various sub-groups of those treated, and will not exclusively focus on a pure control-treatment comparison. For instance, we will evaluate whether the impact on households with working age adults is less than that on households with only elderly people, or with only elderly people and children. To this extent, we have designed an evaluation that will not only address what the effect of the intervention is, but will also allow us to investigate “how” and through what mechanisms its impact on vulnerable populations occurs. Our results will highlight areas for further research into the specific methods through which community-run CCTs can be made more effective.

Despite these encouraging facts, we are aware of the fact that an unexpected change in government could lead to a lower commitment to expanding the community-run CCT program. While this is unlikely since the government has already secured the funds and is committed to beginning the expansion of the program nationwide by the end of 2012, we hope that our affiliations with IFPRI and the World Bank and our involvement in the larger development research, operations, and academic communities will still ensure that our findings are widely disseminated to an audience of policymakers around the world. In this way, even if the findings were of limited use to Tanzania, they would serve a broader community of developing countries. Indeed, IFPRI and the World Bank will work jointly to disseminate the results through publications and learning events, as well as their online presence. Additionally, the lead investigators on the project are committed to presenting the findings at a number of high-profile U.S. and international conferences and policy events that can bring attention to the intervention and its impact. They are also committed to printing and disseminating the findings quickly, and placing them on the internet, so that can be immediately useful to practitioners in development.

>> Read the full Policy Influence Plan from this impact evaluation >>

Micro entrepreneurship support program in Chile: Impact evaluation

In this plan, the team flags the risk that the randomization of the evaluation could be misinterpreted and manipulated by the media, which should be easily mitigated by briefing the media on the nature of the project very early on and providing examples of similar studies in the region and how they have been effectively used to improve the design of development programmes and policies.

We believe risks related to the screening of findings and recommendations to suit political purposes are very small. Indeed, the government itself (FOSIS) has requested that this program be evaluated, with the objective of learning from the results and eventually making modifications to the program design. Additionally, the results of the evaluation will be published through an academic paper and will be largely disseminated in both academic and policy-maker circles.

Similarly, risks related high political turnover are very small, as the program has been implemented since 2006 and is very likely to continue regardless of political turnover. In fact, the evaluation process started with the previous government (of a different political coalition), and has continued flawlessly.

In addition, given that there is a new government in place, “bad” results for the evaluation of PAME could open an opportunity to improve the program by the new administration. On the other hand, if program results are positive, this could be another opportunity for re-launching a similar fine-tuned program. So risks of the government not disseminating results are minimal.

However, we believe there are risks related to the manipulation of information by the press in the case the evaluation design and objectives are not properly understood. In particular, we have considered the risk of a misinterpretation of the randomization design in the pilot program. The fact that there is a control group that will not receive the treatments means there is a risk of the government being criticized and accused of denying the program to certain people. In order to minimize this risk, we have decided jointly with FOSIS to be particularly cautious about disseminating any intermediate results before the impact evaluation is completed. 

>> Read the full Policy Influence Plan from this impact evaluation >>

Resources