Accuracy Type

For classification and regression models, you can choose Accuracy Type. There are two choices:

Maximum Average Accuracy provides the best general approach; there are situations when Maximum Overall Accuracy is a better approach. See Comparison of Accuracy Types for detailed information.

Maximum Average Accuracy

Maximum Average Accuracy is the average per-class accuracy achieved at a specific probability threshold that is greater than the accuracy achieved at all other possible thresholds. The thresholds are choices of positive probability shown in the Receiver Operating Characteristics (ROC) chart. Above the chosen threshold, the model predicts the positive class. Below the threshold, the model predicts the negative class. Per-class accuracy is computed as follows:


Accuracy for class 1 = (number predicted to be class 1)/(number in class 1)
Accuracy for class 0 = (number predicted to be class 0)/(number in class 0)

In terms of the ROC chart,


Accuracy for class 1 = True Positives/( True Positives + False Negatives)
Accuracy for class 0 = True Negatives/( False Positives + True Negatives)

Note that False Negatives are positive class examples that are falsely predicted as negative.

The average per-class accuracy is then computed as the average of the accuracy for class 1 and the accuracy for class 0:


0.5*(Accuracy for class 1)  +  0.5*(Accuracy for class 0)

Maximum Overall Accuracy

Maximum Overall Accuracy is the overall accuracy achieved at a specific probability threshold that is greater than the accuracy achieved at all other possible thresholds. Maximum overall accuracy is the simple accuracy of the model.

Overall Accuracy is the number of correct predictions by the model divided by the total number of instances in the training data.

In terms of the Receiver Operating Characteristics (ROC) chart,


Overall Accuracy = (true positives +  true negatives)/(true positives + true negatives + false positives + false negatives)

Comparison of Accuracy Types

If you select Maximum Average Accuracy, the result is a model whose predictions work well in each class. For example, the model is willing to sacrifice accuracy in a dominant class in order to correctly identify rare class examples. The result is a model whose accuracy, as measured by the average of the per-class accuracies, is highest. Such a model is also known as a proportional errors model, because the model attempts to get the same fraction of errors in each class (and attempts to make that fraction as low as possible).In this case, the user wants to get similar accuracy for each one of the classes. Similar accuracy for each class implies that the user intrinsically assigns different costs to different types of mistakes. That is, making a mistake on a frequent class is not perceived by the user as costly as making a mistake on a less frequent class.

If you select Maximum Overall Accuracy, no such attention is paid to rare classes. An error is an error. The model simply tries to make the overall error rate as small as possible. This works well when all errors are equally important or when the distribution across target classes is relatively uniform. However, if there are rare classes, the model may ignore them, because it must sacrifice accuracy in the dominant class(es) in order to find members of the rare class. In extreme examples (which occur frequently in business domains), the model becomes trivial, predicting only the dominant class in a binary target domain. This is typically useless for applications such as targeted marketing. Maximum overall accuracy is desirable when all errors are treated the same way. In this case the user doesn't care what type of error is made. The goal is to have the smallest number of errors overall.