Error Distribution

Description

This function creates a histogram on the prediction error for one or multiple models.

Application

Typically, you would use this function after a model is trained to check the model performance on an unseen test dataset. With this function you can visualise the distribution of the prediction error. If you have multiple models the error distribution of all models can be easily compared with each other.

How to use

• Select the dataset you want this function to work on in the field Data.

  Make sure that the data contains a column for each output you want to evaluate. Otherwise no prediction error can be calculated and the step will produce an error.

• Select all Models which you want to evaluate in the corresponding field.
• After at least one model was selected the field Outputs will appear. Select one or multiple of the model outputs here for which you want to analyse.

  If multiple models are selected only outputs which are common to all models will be presented. If the models don’t have a common output, the list will be empty and no output can be selected.

• The Max Error option sets the max value of the x-axis of the distribution plot (histogram) that will be generated. The error value is expected in %. That is, the x-axis will span the range [0, Max Error]. If you don’t provide a value, Max Error = 0 will be assumed which will cause an error.

  The number of bins used for the histogram is 80. The size of a bin is determined by Max Error:

  Size of bin = Max Error / 80

• Click Apply to generate the error distribution plot.

Examples

The examples are created for models trained on the composite material properties dataset from Challenge 1 - Composite Materials. The models are trained to predict the ultimate strain of a composite.

The first example shows the error distribution for a single model (Neural Network). Max Error is set to 10. That is, the histogram is created for errors in the range [0, 10] %. As the bars towards the right side are still quite high this means we have cut off too early.

The second example shows the error distribution for the same model, but this time Max Error is set to 40. The distribution towards the right side now tends towards 0 which means the relevant error range is captured.

As we changed Max Error the bin size also changed. In this second example one bin represents a much larger error range and therefore the counts are much higher compared to the first example.

The next example shows the same model (Neural Network) with Max Error set to 40 in comparison to a second model (Polynomial Regression). Both models predict the same output. Therefore, they can be compared with each other. The two distributions are plotted in different colors and with some transparency.

As you can see the error distribution of the two models shown in that plot are quite different. The neural network is closer to the ideal distribution with a high count of small errors and a distribution which fades out quickly. The polynomial model with the slowly diminishing count is much worse as the counts are nearly twice as high on large errors (15 … 30%).

More on this step

The error distribution can be integrated, and the integrated curve can be visualised using the step Compare Against Criteria.

This is a special form of distribution plot for model error evaluation. You could also use Dataset prediction, include prediction error in the prediction, and apply the Distribution step to the error columns to achieve similar plots.

How to read the plot

• On the x-axis the prediction error is plotted as Absolute Relative Error in %.
• The range of the x-axis is [0, Max Error].
• The number of bins is set to 80.
• On the y-axis the count for each bin will be plotted.