Comparative assessment of the measures of thematic classification accuracy

Abstract

Accuracy assessment of classified imagery is an important task in remote sensing. Various measures have been developed to describe and compare the accuracy of maps and the performance of different classifiers, but the extent to which these measures are consistent with each other is largely unknown. In this paper the consistency of fourteen category-level and twenty map-level accuracy measures was tested on 595 published error matrices using nonparametric correlation coefficients (Spearman's "rho" and Kendall's "tau-b") as well as the probability of concordance. The results show that four groups can be identified for the category-level measures and three groups for map-level measures. The consistency among the measures within a group is generally higher than that among the measures from different groups though all the measures at the same level are highly consistent with each other. We recommend that user's accuracy and producer's accuracy and the overall accuracy should be provided as primary accuracy measures and the two relative entropy change measures and the mutual information normalized by the arithmetic mean of the entropies on map and ground truthing be provided as supplementary measures. The chance-corrected, error matrix-normalized and user's and producer's accuracy-combined measures were found to contain estimation and interpretation problems at both category- and map-levels and are therefore not recommended for general use.