9. Evaluation and learning
This final step is about mapping the key indicators to measure the success of your policy influence activities. The indicators need to be consistent with the policy influence objective that you set. The indicators should not be limited to output level.
Note: 3ie requires that funded research teams submit information on the number of people reached by different dissemination activities, and documentation of adoption of study recommendations. This information is part of 3ie’s own reporting to its donors, and so helps ensure future funding for studies.
You can use your Theory of Change to help frame your indicators by looking at the interventions and activities that will help bring about the change you are proposing. So, if you say media engagement will play an important role as part of your overall strategy you should have indicators set that monitor your level of media engagement and coverage (i.e. media clippings/citations). Look at the interventions/activities across your Theory of Change and see if you can assign SMART indicators to them.
The team should identify: Level 1. Research dissemination (e.g. publishing, media coverage, web downloads, knowledge sharing/dissemination events, # of people reached/consulted, # of high level policy makers engaged etc)
Level 2- Research communication or evidence that people are not just downloading the study but that the study is catalysing a debate (e.g citation indices, people reporting they have been influenced by the study, feedback ratings etc.)
Level 3- Research uptake or evidence of practical use (e.g. adopted legislation, behavioural changes in practices on the ground etc).
Level 4 - Research impact or practical evidence of use of research, change in behaviour of policy. This will be difficult to attribute and track in the lifetime of the project
The section on measuring and learning identified an interesting way of measuring the influence of its policy brief by providing feedback ratings “returned without interest”, “Noted”, “put on reading list”,….and “immediately adopted into policy”.
China has a system of giving immediate feedback to scholars and policy analysts after the submission of policy briefs … The State Council and provincial government authorities will provide written feedback, including what was done with the brief (“returned without interest” / “noted” / “put on reading list” / “discussed and contents noted” / “sent to ministries and others for further policy discussion and feedback” / “sent to ministries and others for action” / “immediately adopted into policy”.
We will also follow the upscaling of the activities. Our plan is to have the provincial government begin to provide vitamins with the egg and milk project. Our ultimate influence will be the number of counties in which vitamins are given and the number of students in each county.
If we can convince them (negotiations in progress) to upscale in a phased roll-out format, we can compare school year-end standardized test rankings.