Brian Belcher reports on PORIA Workshop
I attended the “Workshop on Best Practice Methods for Assessing the Impact of Policy Oriented Research” (November 11–12, 2014) hosted by the International Food Policy Research Institute (IFPRI) in Washington DC, and co-sponsored by IFPRI, the Policy, Institutions and Markets Consortium Research Program (PIM) and the Standing Panel on Impact Assessment (SPIA) of the Consultative Group on International Agricultural Research (CGIAR). Lots of acronyms!
The workshop was organized with the recognition that methods and practice for evaluating the impact of policy-oriented research (POR) lag most other research sectors. Policy-oriented research was defined as research aimed to result in new or improved policies, regulations, institutions (and their management) that enhance economic, social and environmental welfare. Although some types of POR can be subjected to rigorous impact assessment (IA) using randomized trials and econometric assessments, most such research is not amenable quantitative assessment. The workshop aimed to:
1. Define the scope of POR within the context of the CGIAR and the challenges this poses for IA.
2. Take stock of what we already know about PORIA in terms of methods, case studies, etc. and of promising new developments in terms of approaches and methods. Identify the key challenges for conducting PORIA and the knowledge gaps that need to be filled.
3. Develop a typology of POR that can help differentiate methods and approaches that are relevant in different cases. Possible criteria to consider are type of POR, type of policy process, type of audience. Present good practice/current state of the art for IA in the different types.
4. Explore implications for the selection of intermediate indicators that can help ensure that POR is on track to have impact, and which can contribute to credible evidence bases for subsequent ex post assessments.
5. Prepare a clear statement that defines realistic expectations for what can and cannot be achieved in evaluating the impact of different types of POR. The donor community was seen as a primary audience.
There were over 40 participants, including academics, donors and CGIAR scientists. A majority of participants were economists, many with extensive experience doing research on the impact of policy – that is, estimating ex ante or quantifying ex post the welfare benefits of a policy or policy change. While this is a fundamental part of the problem, the more challenging question conceptually and technically is how to measure the influence of research on the policy process itself. Of all publicly-funded activities, research may be the most difficult to effectively monitor and evaluate because the path from research to impact is long and indirect, especially in complex systems.
Overall, there was a high level of consensus that we need to think in terms of contribution not (sole) attribution, that theory based approaches and plausible causal stories may often be the best possible, and that we need to “think like lawyers” in terms of assembling evidence to make a case. The donors present (ACIAR, USAID) expressed their appreciation for the effort to improve evaluation of this kind of research – they need to be able to demonstrate value for money to fund this kind of research in the face of many competing demands - and they supported the general consensus. I was able to report that the UK’s DfID has also supported this approach in a current project on “Improving the way knowledge on forests is understood and used internationally”
The problem and the general approach were very much in line with the design and the approach developed by the research program of the Canada Research Chair on Sustainability Research Effectiveness. The workshop sponsors used a more restricted definition of “Policy-oriented research (POR)” and the workshop emphasized government policy as the object. The CRC program is interested more broadly in evaluating research that aims to inform policy and practice among a wide range of individuals, organizations and governments at different levels. The typology proposed and discussed at the workshop classified research by scale and thematic focus, and was therefore somewhat limited in its application. I proposed an alternative classification based on type of outcome, which is more in line with the approach we have developed with the CRC program and at the Centre for International Forestry Research (CIFOR for those who are making lists of abbreviations). I was able to share and get feedback on the concepts and approaches we have developed over the past few years.
Meeting information, background documents and presentations are available at: https://sites.google.com/a/cgxchange.org/poria-workshop/home
I will be happy to discuss further with anyone who is interested in these issues.