Gaps in Evaluation methods in development, and what innovations might address these gaps
Please join us for a morning of talks and a discussion addressing the “Gaps in Evaluation methods in development, and what innovations might address these gaps”. Refreshments will be provided.
The speakers (Audrey Prost, Nancy Cartwright, and Macartan Humphreys) are leaders in the fields of political science, philosophy of science, and social science, and will be presenting their perspectives on this issue. Short biographies are included below.
Audrey Prost
9:30 – 10:30 am
Using mixed methods to evaluate the effects of scaling up participatory women’s groups for maternal and newborn health in rural India
This talk will describe the use of mixed and multiple methods to evaluate the effects of participatory women’s groups to improve maternal and newborn health across the State of Jharkhand, in eastern India. We will use this case study to outline key challenges faced when mixing methods, and discuss the practical and epistemological implications of solutions suggested by the MRC guidance for process evaluation, Program Impact Pathways adepts, and approaches to mixed methods synthesis.
Bio: Audrey works at UCL’s Institute for Global Health and in close collaboration with the Indian civil society organisation . Together they have conducted two large cluster randomised controlled trials of participatory women’s groups to improve maternal and newborn health in largely tribal areas of Jharkhand and Odisha, eastern India. The Ekjut trial, which was published in the Lancet in 2010, won the Society for Clinical Trial’s Award of ‘Trial of the Year’ in March 2011. Audrey, Ekjut, and the Public Health Foundation of India now collaborate on CARING, an MRC/Wellcome Trust/DFID-funded trial testing a community intervention with participatory women’s groups and home visits to improve the growth of children under two in Jharkhand and Odisha. Audrey is also an associate editor of Trials, and interested in the design and analysis of trials of complex social interventions.
Nancy Cartwright
10:45 – 11:45 am
Intervention-centring, Context-centring: Two approaches, two sets of method gaps
The intervention-centred approach to predicting policy effectiveness pictures successful policies as having something like an internal capability to produce the outcome if only circumstances are right. The big question then is, ‘When are circumstances right?’ Hence the current emphasis on ‘What works when and where?’ The problem is that we don’t know how to answer this question. For instance subgroup analysis can’t do the job. That’s not because it is open to data mining nor because it cannot establish causality; rather, it is looking to identify moderator variables for the theory of change, whereas what’s needed is to identify what systems will afford the intervention/outcome process envisaged in the theory of change. The context-centred approach essentially requires the building of a local causal model. Although there may be much discipline-specific expertise at this, there is little general guidance for how to build such models, for how to categorise the variety of different kinds of evidence that can bear on them, or for how to put the evidence together to arrive at an overall judgment.
Bio: Nancy Cartwright is Professor of Philosophy at the Department of Philosophy, University of Durham and at the University of California, San Diego (UCSD). Her research interests include philosophy and history of science (especially physics and economics), causal inference, causal powers, scientific emergence and objectivity and evidence, especially for evidence-based policy [EBP]. Her current work, for the project , investigates how to use scientific research results for better policies. She is a member of the UK voluntary research network , which ‘aims to develop a new methodology for policy formulation, deliberation, evaluation and choice.’ She has worked with others on projects in this area on education, child protection and international development.
Macartan Humphreys
11:45 – 12:45 pm
Mixing Methods: a Bayesian Approach
This talk will present a new approach to multi-method research that generates learning from quantitative and qualitative evidence. The method uses a Bayesian framework to allow causal inferences from combinations of correlational and process-level observations. In addition to posterior estimates of causal effects, the framework updates confidence in the assumptions underlying the model. The approach will be illustrated with applications to substantive issues in political science. Finally, there will be a demonstration of how the framework can yield guidance on multi-method research design.
Bio: Macartan works on the political economy of development and formal political theory. Ongoing research focuses on post-conflict development, ethnic politics, political authority and leadership, and democratic development with a current focus on the use of field experiments to study democratic decision-making in post-conflict and developing areas. He has worked in Chad, Ghana, Haiti, Indonesia, Liberia, Mali, Sao Tome and Principe, Sierra Leone, Senegal, Uganda, and elsewhere. He is a former Trudeau fellow and scholar of the Harvard Academy, a research fellow at the WZB Berlin, the Executive Director of the and a Professor of Political Science at Columbia University.
