EvalC3 Online: Introduction

Background

The design of EvalC3 Online makes use of two sets concepts and methods:

• Qualitative Comparative Analysis, a body of methods developed in Political Science. Especially its view of causality (equifinality, asymmetry, conjunctural) and the importance of combining cross-case and within-case analysis)
• Supervised machine learning, a body of methods used for the development of predictive models. Especially what is known about different search algorithms and how the performance of prediction models generated by these algorithms can be evaluated.

Overall purpose

EvalC3 Online enables users:

1. To identify sub-sets of attributes that describe an intervention & its context, and which are good predictors of the achievement of an outcome of interest.
2. To compare and evaluate the performance of these predictive models.
3. To identify relevant cases for follow-up within-case investigations to uncover any causal mechanisms at work.

Workflow

The worksflow diagram below shows the main sequence of steps that would normally be followed, when analysing a set of of data. Recursion and reiteration of earlier steps is also likely to be common.

Range of uses

EvalC3 can be of use at all stages of a project cycle:

During project selection

• To identify what attributes of project proposals are the best predictors of whether a project will be chosen for funding, or not
• To identify how well a project proposal appraisal and screening process is as a predictor of the subsequent success of projects in achieving their objectives

During project implementation

• When the effectiveness of specific activities are being measured using survey instruments which include both specific (facet) and general (global) measures of satisfaction with service or product delivered. EvalC3 can identify what combination(s) of facets best predict global satisfaction. For example:
• Participants experiences with workshops and training events
• Donors and grantees experiences of their working relationships with each other

During a project evaluation

• “Causes of effects” analysis: To identify what combination(s) of project activities (and their contexts) were associated with a significant improvement in beneficiaries lives.
• “Effects of causes” analysis: To identify what combinations of improvements in beneficiaries lives were associated with a specific project activity (or combination of)
• To identify “positive deviants” – cases where success is being achieved despite the fact that failure is the most common outcome.

During a review of existing evaluations

• Re-analysing data that was collected, to verify the results. This is often possible with QCA based evaluations because QCA data sets are usually published as annexes to evaluations
• Synthesising the results of multiple evaluations, into prediction rules concerning different types of outcomes

For more information on each of the stages in the flow chart above, see How to use EvalC3

Acknowledgements

EvalC3 Online was built by Reyt, based on the Excel app EvalC3, built by Mark Skipper at Aptivate, as designed by Dr Rick Davies.

With funding provided by Sports England, managed by a team led by Dr Katie Shearn at the Sheffield Hallam University Advanced Wellbeing Research Center