About the confusion-matrix-based assessment of the results of imbalanced data classification

When applying classifiers in real applications, the data imbalance often occurs when the number of elements of one class is greater than another. The article examines the estimates of the classification results for this type of data. The paper provides answers to three questions: which term is a more accurate translation of the phrase "confusion matrix", how preferable to represent data in this matrix, and what functions to be better used to evaluate the results of classification by such a matrix. The paper demonstrates on real data that the popular accuracy function cannot correctly estimate the classification errors for imbalanced data. It is also impossible to compare the values of this function, calculated by matrices with absolute quantitative results of classification and normalized by classes. If the data is imbalanced, the accuracy calculated from the confusion matrix with normalized values will usually have lower values, since it is calculated by a different formula. The same conclusion is made for most of the classification accuracy functions used in the literature for estimation of classification results. It is shown that to represent confusion matrices it is better to use absolute values of object distribution by classes instead of relative ones, since they give an idea of the amount of data tested for each class and their imbalance. When constructing classifiers, it is recommended to evaluate errors by functions that do not depend on the data imbalance, that allows to hope for more correct classification results for real data.

1. Galar M., Fernandez A., Barrenechea E., Bustince H., Herrera F. A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 2012, vol. 42, no. 4, pp. 463–484. https://doi.org/10.1109/tsmcc.2011.2161285

2. Miller G. A., Nicely P. E. An analysis of perceptual confusions among some English consonants. The Journal of the Acoustical Society of America, 1955, vol. 27, no. 2, pp. 338–352. https://doi.org/10.1121/1.1907526

3. Brodersen K. H., Ong C. S., Stephan K. E., Buhmann J. M. The balanced accuracy and its posterior distribution. International Conference on Pattern Recognition, 2010, Istanbul, pp. 3121–3124.

4. Starovoitov V., Golub Yu. New function for estimating imbalanced data classification results. Pattern Recognition and Image Analysis, 2020, vol. 30, no. 3, pp. 295–302.

5. Hand J. D., Till J. R. A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine Learning, 2001, vol. 45, no. 2, pp. 171–186. https://doi.org/10.1023/A:1010920819831.

6. Fawcett T. Using rule sets to maximize ROC performance. Proceedings IEEE International Conference on Data Mining, 2001, pp. 131–138. https://doi.org/10.1109/ICDM.2001.989510

7. Ballabio D., Grisoni F., Todeschini R. Multivariate comparison of classification performance measures. Chemometrics and Intelligent Laboratory Systems, 2018, vol. 174, pp. 33–44. https://doi.org//10.1016/j.chemolab.2017.12.004

8. Shhetinin E. Ju., Sevast'janov L. A., Demidova A. V., Kuljabov D. S. Klassifikacija povrezhdenij kozhi po dannym dermatoskopii s ispol'zovaniem metodov glubokogo obuchenija [Dermatoscopy classification of skin lesions using deep learning methods]. Matematicheskaja biologija i bioinformatika [Mathematical biology and bioinformatics], 2020, vol. 15, no. 2, рр. 180–194.

9. Buscema P. M., Massini G., Maurelli G., Artificial Adaptive Systems to predict the magnitude of earthquakes. Bollettino di Geofisica Teorica ed Applicata, 2015, vol. 56, no. 2, pp. 227–256.

10. Valverde-Albacete F. J., Pelaez-Moreno C. 100 % classification accuracy considered harmful: The normalized information transfer factor explains the accuracy paradox. PloS One, 2014, vol. 9, no. 1, р. e84217. https://doi.org/10.1371/journal.pone.0084217