ICML-98 Submission #127

Title: A Randomized ANOVA Procedure for Comparing Performance Curves

Authors: Justus H. Piater, Paul R. Cohen, Xiaoqin Zhang, Michael Atigetchi

		Computer Science Department
		University of Massachusetts
		Amherst, MA 01003

Abstract

  Three factors are related in analyses of performance
  curves such as learning curves: the amount of training, the learning
  algorithm, and performance.  Often we want to know whether the
  algorithm affects performance and whether the effect of training on
  performance depends on the algorithm.  Analysis of variance would be
  an ideal technique but for carryover effects, which violate the
  assumptions of parametric analysis of variance and can produce
  dramatic increases in Type I errors.  We propose a novel, randomized
  version of the two-way analysis of variance which avoids this
  problem.  In experiments we analyze Type I errors and the power of our
  technique, using common machine learning datasets.

Keywords: Performance curves, analysis of variance,
          randomization, TypeI error, power

Email address of contact author: piater@cs.umass.edu

Phone number of contact author: (413) 545-3492