EvalC3 Online: How to use the Design and Evaluate screen

What you can see

Thsi screen is the most central part of EvalC3 online. It has five distinct sections, that fall into three groups:

1. The Model Design "menu" on the top left, which shows:
   - whether the expected outcome is present or absent
   - which case attributes are expected to be present, absent or not relevant
2. Information on model performance, on the right, which includes:
   - Confusion Matrix, which describes the overall performance of the model in the Model design menu, in raw numbers of cases
   - Model Status, which summarises that performance in terms of its necesisty and or sufficiency
   - Model Performance, which details that performance using a range of measures suitable for diferent kinds of contexts. A seperate Help page will soon be developed for this section
3. Search for new models a drop down menu listing options for the automated i.e. algorithm-based search for best performing predictive models. See this help page for more details. But see also the short note at the bottom of this page.

What you can do

A suggested workflow

1. Start by manually designing a predictive model. In the "Model Design" section select the attributes that you think will predict the presence of the outcome of interest.

2. Then inspect the Confusion Matrix on the right , to see how well it performs.
   A good model will have a high proportion of all cases in the True Positive (TP) and True Negative (TN) categories.

3. Then inspect the Model Status panel immediately below, to see if the model attributes are necessary and or sufficient for the outcome to occur. Click on the Information balloons to see how these are identified

4. Then inspect the "Model Performance" measures further below to get a more detailed perspective on the performance of the model.
   Click on the Information balloons to see how each of these are measured

5. Then under "Search for new models" click on any of the buttons to generate alternative models using four different search algorithms

6. Revisit the "Design" section to see what has been found and inspect its performance
   Then tweak the features of the model, to see what differences various small changes make to its performance.

A note on "Search for new models"

Two approaches can be taken to finding good predictive models

• Manual hypothesis-led inquiry
• Algorithm-based searches

Four types of search algorithms are available within EvalC3 Online:

1. To find the single best predictive model
   .....For quick answers
   .........Cumulative single attribute searches
   ....To avoid the “local optimum" problem
   ........Exhaustive searches of multiple attribute combinations
   ........Evolutionary searches, using a genetic algorithm.
2. To find the best set of predictive models, covering all observed outcomes
   ........Decision Tree searches.