Predictive Confidence is a number between 0 and 1 that indicates how much better the predictions made by the tested model are than predictions made by a naive model. The naive model always predicts the mean for numerical targets and the mode for categorical targets.
The following formula defines Predictive Confidence:
Predictive Confidence = MAX((1-((error of model)/(error of naive model))), 0)
For a categorical target, the error for the naive model is (n-1)/n
, where n
is the number of distinct values of the target attribute. For example, suppose we are trying to predict a categorical target that has values 0 and 1. Suppose the distribution of the target attribute is 95% 0 and 5% 1. In this case, the naive model always predicts 0 and is correct 100% of the time for target = 0; it is incorrect is 100% of the time for target = 1. On average, across the target values, the naive model is an error of (0% + 100%)/2 = 50%. If Predictive Confidence is greater than 0, then the tested model is better than the naive model. There is a similar analysis for numerical values.
If Predictive Confidence is 0, the model's predictions are no better than using a naive model. If Predictive Confidence is 1, the predictions are perfect. If Predictive Confidence is 0.5, the model has cut by 50% the error that you get using a naive model.
Copyright © 2006, 2008, Oracle. All rights reserved.