Day 1-Session 1

10.00-10.30

What is policy evaluation?

·   Evaluation and the policy process

·   Monitoring vs evaluation

·   Impact and process evaluation

·   Different Meanings of Impact

·   Quantitative and qualitative evaluation

Group Q/A and discussion  

10.30-10.40

10.40-11.00

11.00-11.15

Evaluating the Problem – Reducing drug-related crime

·   Situation Analysis

·   What is the nature, magnitude and causes of the problem at hand?

Group Work:

·   Problem-Tree analysis – identifying causes, consequences and interacting components. How complex is the complexity of the problem?

Feedback from group work

11.15-11.30

11.30-11.45

11.45-12.00

Identifying Policy Options

·    Evaluating existing policy levers – are they working?

·    What has worked elsewhere? -  Using existing evidence

·    Stakeholder involvement

Group Work:

·   Identifying policy options for reducing drug-related crime

Feedback from group work

12.00-13.00

Lunch

Day 1-Session 2

13.00-13.15

13.15-13.40

13.40-14.00

Establishing a Theory of Change – Dealing with Complexity

·  How is a policy supposed to work?

·  Inputs, activities, agencies, outputs, outcomes

·    Evaluating policy effectiveness at different levels– individual, social, physical environment, macro-sectorial.

·  Testing assumptions using quantitative and qualitative methods/data

Group work:

· Drug courts – building a theory of change for how drug courts are supposed to work

Feedback from group work

14.00-14.20

14.20-14.40

14.40-15.00

Developing a Monitoring and Evaluation Framework

·    Selecting indicators

·    Developing a monitoring and evaluation matrix

·    Data transfer – from Excel to statistical software

·    Developing and monitoring delivery trajectories

Group work:

What indicators might be used to monitor and evaluate the effectiveness of drug courts?

Feedback from group work

Day 2- Session 1

10.00-11.00

11.00-11.30

11.30-12.00

Counterfactual Impact Evaluation

·  What is ‘counterfactual’ evaluation?

·  Interrupted Time Series design

·  Matched comparison design (Propensity Score Matching)

·  Difference-in-Differences

·  Regression Discontinuity Design

·  Randomised Controlled Trials

Group Work:

In small groups participants will discuss case studies of each of the counterfactual impact evaluation methods above. These case studies will have been distributed electronically to be read before the training.

Feedback from group work:

What have participants learned about the use of counterfactual impact evaluations for policy evaluation?

12.00-13.00

Lunch

Day 2-Session 2

13.00-13.30

13.30-13.50

13.50-14.00

Process Evaluation

·   What is process evaluation?

·   Evaluating the fidelity and quality of implementation

·   Clarifying causal mechanisms

·   identifying contextual factors affecting implementation and outcomes

Group Work:

In small groups participants will discuss case studies of process implementation. Case studies will have been distributed electronically to be read before the training.

Day 2 Session 2

14.00-14.30

14.30-14.45

14.45-15.00

Evidence Synthesis

·  Harnessing what is already known

·  Different types of research synthesis

·  The structure of a systematic review

·  Examples of evidence synthesis for policy making

·  Sources of sound evidence from systematic reviews

Group work:

In small groups participants will discuss the potential for using evidence synthesis in their policy evaluation work.

Feedback from group work