Evidence shows that political and institutional context issues are the most important set of factors affecting both the timing and design of your evaluation and the opportunities for the findings to influence policy. Understanding the contextual factors that affect the relevance of your study is essential to help you think through and plan your policy engagement. The context analysis should focus on the policy relevance and timeliness of the evaluation. It should also highlight the opportunities and challenges that will affect the dissemination and uptake of the evaluation results.
Some of the questions you may want to address in this section are: What institutional, cultural, or political factors will affect whether and how you can shape change? What is the culture of use of evidence in policy making in the country in general, and in the sector of your study? What are the knowledge gaps that the study is trying to address? Is there demand from policymakers for the kind of evidence the evaluation will provide?
Louise Shaxson from ODI talks about the importance for researchers to have a detailed knowledge of the policy and political context.
The following example of a context analysis provides very good insights on the political process, key players and critical factors in youth unemployment in South Africa. There is a clear demand from the Government and a strong signal in favor of evidence base policy.
Unemployment, especially youth unemployment, is a key policy (and political) challenge for South Africa. At least 3 million young people are classified as unemployed – with many more discouraged and not looking for work. In an attempt to tackle this, and other economic challenges, the South African government has proposed a number of economic strategies, the most recent being the New Growth Plan (NGP) announced in late 2010. This plan emanated from the newly created Ministry of Economic Development and targets the creation of 5 million new jobs by 2020. The NGP proposes to create many of these jobs through investment in infrastructure and other government led initiatives. President Zuma endorsed the plan at the African National Congress’s (ANC’s) 99th birthday celebration in January 2011 and has indicated that job-creation would be top of the cabinet’s agenda in 2011. These developments indicate that reducing unemployment is a priority for the South African government.
In parallel to this increased government focus on unemployment in the last five years there has been an increase in youth nationalism and voice that has coalesced around the ANC Youth League Chairman Julius Malema. Malema’s constituents are those that have been marginalised by South Africa’s economic transformation, including unemployed youth. Malema’s rhetoric has been fierce and confrontational and there has been very little admonishment from more senior members of the ANC. The rise of this faction within the ANC adds additional importance to the creation of jobs which may help to placate a politically, and economically, destructive constituency.
The broad church nature of the ANC and its participation in the Tripartite Alliance (the ANC, Congress of South African Trade Unions and the South African Communist Party) means that the real political economy struggles happen within the ruling party. These often happen behind closed doors and thus it is often difficult to know which faction is in the ascendancy, or in fact who belongs to which faction. What this means in terms of influencing policy is that having a policy proposal associated with a particular faction may be detrimental. It also suggests that concrete evidence for the success of a proposed intervention and broad knowledge of this, through the media and meetings with stake-holder groups, is essential in order to create support and an environment conducive for the implementation of the policy.
The agencies that actually implement policy are the various South African government departments and these are the primary stakeholders that need to be engaged with for successful implementation of the policy. The main departments which will be impacted by a youth wage subsidy are: National Treasury; Trade and Industry; Labour; Economic Development; and the Presidency. This project is working directly with the National Treasury but will need to obtain support from these various other departments. Most are currently represented on the steering committee and will need to be engaged more fully.
Both the government as a whole and National Treasury are supportive of evidence based policy making. A specific Department of Performance Management, Monitoring and Evaluation within the Presidency has been created under President Zuma. There is now pressure on individual department to monitor and evaluate performance and policies. National Treasury has also recently been building up their understanding of impact evaluation, particularly through randomised trials, by sending employees to attend workshops run by organisations such as J-PAL.
The context analysis provides useful insight on the specific factors, opportunities and challenges that will affect the dissemination and uptake of the evaluation results. The project is also incorporating some intermediate evaluation to respond to the demand of policymakers. It also acknowledges the different level of policy influence needed and sequencing needed between local and national policy work.
The environment in China today is mostly positive – especially with regards to the project that we plan to evaluate: the impact of vitamin supplements on anemia and educational performance in schools.
First, there are government fiscal resources available for social policy promotion and problem solving. Second, most officials – national, regional and local – are professionals. They have degrees (top level officials often have graduate degrees) in education and/or specialty fields. Third, officials that do innovative programs that work are more apt to be considered for promotion. Therefore, there is an alignment between our evaluations—i.e. showing whether a new program works—and the demands for such information (for personal gains of officials and for the ability to carry out effective policy for those that want to better execute their job).
These demands, though, mean that there is a need to carry out evaluations that can yield feedback as soon as possible. One year is a long time for officials. It is hard to interest them in an evaluation in which the results can not be seen for three or more years.
Therefore, it is imperative to make sure there are intermediate evaluations. This is why we plan to do a baseline, an intermediate evaluation and a final evaluation.
The nature of China’s education policy also means that policy work has to address both national, regional and local policy maker needs. If a local official wants to innovate, he often needs (or would like) the “cover” of a statement or broad initiative of a national policy maker. Therefore, we find it is important to get upper level officials to condone a policy direction (e.g., support nutrition programs inside schools) and then allow local officials to experiment and evaluate these programs.