Details

Category: Policy, Projects & Programmes

Published: Tuesday, 27 May 2014 11:38

Written by: Matt Pye - HWD

Overview

This two-day training course provides participants with a comprehensive understanding of policy evaluation and the various types of evidence used in the evaluation process. On Day One, participants will explore the concept of policy evaluation, its importance, and its role in evidence-based policy. They will delve into establishing a theory of change, understanding different types of evidence such as systematic reviews, surveys, qualitative methods, interrupted time series analysis, and more. Day Two focuses on additional types of evidence, including single group before and after designs, matched comparison designs, difference-in-difference analysis, regression discontinuity design, randomised controlled trials, implementation evaluation, and economic appraisal. The course also covers practical aspects of presenting evaluation evidence, including structuring reports, writing in non-technical language, and delivering key messages to policy makers. By the end of the training, participants will have a solid foundation in policy evaluation and be equipped with the knowledge and skills to conduct robust evaluations and effectively communicate their findings to stakeholders.

Learning Objectives

1. Understand the importance and purpose of policy evaluation, including its role in evidence-based policy making and decision-making processes.

2. Identify and differentiate between various types of evidence used in policy evaluation, such as systematic reviews, surveys, qualitative methods, interrupted time series analysis, matched comparison designs, regression discontinuity design, randomised controlled trials, implementation evaluation, and economic appraisal.

3. Gain proficiency in establishing a theory of change and developing logical models that depict causal chains for policy interventions.

4. Evaluate the strengths, limitations, and potential biases associated with different types of evidence used in policy evaluation, enabling informed decision-making regarding their suitability for specific evaluation contexts.

5. Acquire practical knowledge and skills in data collection methods, including surveys, census data, administrative data, qualitative methods (e.g., interviews, focus groups), and observational techniques, for conducting policy evaluations.

6. Develop effective communication strategies for presenting evaluation evidence to policy makers, including structuring reports, writing in non-technical language, and delivering key messages that address policy background, problem description, policy options, implementation issues, resource implications, and recommendations.

 

Course Agenda

Day One

Policy Evaluations – What Are They and Why Do We Need Them

• What do participants understand by policy evaluation?
• This two-day training course provides participants with a comprehensive understanding of policy evaluation and the various types of evidence used in the evaluation process. On Day One, participants will explore the concept of policy evaluation, its importance, and its role in evidence-based policy. They will delve into establishing a theory of change, understanding different types of evidence such as systematic reviews, surveys, qualitative methods, interrupted time series analysis, and more. Day Two focuses on additional types of evidence, including single group before and after designs, matched comparison designs, difference-in-difference analysis, regression discontinuity design, randomized controlled trials, implementation evaluation, and economic appraisal. The course also covers practical aspects of presenting evaluation evidence, including structuring reports, writing in non-technical language, and delivering key messages to policy makers. By the end of the training, participants will have a solid foundation in policy evaluation and be equipped with the knowledge and skills to conduct robust evaluations and effectively communicate their findings to stakeholders.
• What is evaluation and why is it essential?
• Policy evaluation and evidence-based policy
• What factors affect how a policy should be evaluated?
• Evaluation as part of the policy cycle
• The Magenta Book

Types of Evidence - Establishing a Theory of Change

• Activities, Outputs, Outcomes
• Sequencing a logic model – building a causal chain
• Data required for a theory of change
• Case Study

Types of Evidence – Systematic Reviews and Meta-Analysis

• Harnessing what is already known
• Different types of research synthesis
• The structure of a systematic review
• Examples of Statistical Meta-Analysis for Policy Making
• Sources of sound evidence from systematic reviews

Types of Evidence for Policy Evaluation – Surveys, Census Data and Administrative Data

• Using existing survey data
• Collecting survey data Using census data
• The use of administrative data
• Risk of bias with surveys, census data and administrative data

Types of Evidence for Policy Evaluation – Qualitative Methods and Data

• In-Depth interviews
• Group interviews and focus groups
• Other consultative methods
• Observational and Participant –Observation methods
• Ethnography
• Case Studies
• Quality of Qualitative Evaluation
• The contribution of qualitative data to policy making

Establishing a Strong Counterfactual

• The importance of the counterfactual for policy making
• Identifying policy options
• Establishing the net effect of a policy

Types of Evidence for Policy Evaluation – Interrupted Time Series Analysis

• The logic of Interrupted Time Series Analysis
• Examples of Interrupted Time Series Analysis
• Risk of bias with Interrupted Time Series Analysis
• Examples of Interrupted Time Series Analysis in policy evaluation

Day Two

Types of Evidence for Policy Evaluation – Single Group Before and After Designs

• The logic of a single group before and after design
• Examples of a single group before and after design
• Risk of bias with a single group before and after design

Types of Evidence for Policy Evaluation – Matched Comparison Designs

• The principles of matching
• Propensity Score Matching
• Examples of Propensity Score Matching for policy making
• Risk of bias with Propensity Score Matching

Types of Evidence for Policy Evaluation – Difference-in-Difference Analysis

• Single difference and double difference
• Risk of bias with difference-in-difference analysis
• Examples of difference-in-difference analysis in policy making

Types of Evidence for Policy Evaluation – Regression Discontinuity Design

• The logic of regression discontinuity design
• Examples of regression discontinuity design in policy making
• Risk of bias with regression discontinuity design

Types of Evidence for Policy Evaluation - Randomised Controlled Trials

• How RCTs establish a strong counterfactual
• Risk of bias with RCTs
• Examples of RCTs in policy making

Types of Evidence for Policy Evaluation – Implementation Evaluation

• Delivery Trajectories
• Developing a Delivery Plan
• Undertaking a Priority Review

Types of Evidence for Policy Evaluation – Economic Appraisal

• The Green Book
• Cost-Benefit, Cost-Effectiveness and Cost-Utility Analysis
• Monetising cost and benefits
• Establishing New Present Value and Equivalent Annual Net Cost to Business

Presenting Evaluation Evidence

• The 1:3:25 format
• Writing clearly in non-technical language
• The Title
• Key messages for policy makers (one page)
• Executive Summary (three pages)
• Policy background and context
• Problem description
• Policy options
• Implementation and delivery issues
• Resource implications
• Conclusions and recommendations