ICML-98 Submission #172

Building the Case Against Accuracy Estimation 
for Comparing Induction Algorithms

Foster Provost and Tom Fawcett
Bell Atlantic Science and Technology
400 Westchester Avenue
White Plains, New York 10604

Ron Kohavi
Data Mining and Visualization
Silicon Graphics Inc. M/S 8U-876
2011 N. Shoreline Blvd.
Mountain View, CA. 94043

Abstract

We question the common use of classification accuracy to compare
classifiers on natural data sets.  To investigate whether potential
justifications are valid we use ROC analysis, providing a thorough
investigation using standard machine learning algorithms and standard
benchmark data sets.  The results of the study raise concerns about
the use of accuracy for comparing classifiers, even when predictive
performance is the only criterion.  The contribution of this paper is
two-fold.  We analyze carefully a common assumption of machine
learning research, provide interesting insights regarding its
applicability, and discuss their implications.  In doing so we
describe what we believe to be a superior methodology based on ROC
curves for the evaluation of induction algorithms.

Keywords: Inductive learning, classification, accuracy estimation,
ROC curves, cost-sensitive learning, performance analysis, real-world

Contact author: foster@basit.com
Phone: (914) 644-2169

(Contact fawcett@basit.com, 914-644-2193 between 2/28 and 3/9)