Model selection via the AUC

Saharon Rosset

Model selection via the AUC

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We present a statistical analysis of the AUC as an evaluation criterion for classification scoring models. First, we consider significance tests for the difference between AUC scores of two algorithms on the same test set. We derive exact moments under simplifying assumptions and use them to examine approximate practical methods from the literature. We then compare AUC to empirical misclassification error when the prediction goal is to minimize future error rate. We show that the AUC may be preferable to empirical error even in this case and discuss the tradeoff between approximation error and estimation error underlying this phenomenon.

Original language	English
Title of host publication	Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004
Editors	R. Greiner, D. Schuurmans
Pages	703-710
Number of pages	8
State	Published - 2004
Externally published	Yes
Event	Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004 - Banff, Alta, Canada Duration: 4 Jul 2004 → 8 Jul 2004

Publication series

Name	Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004

Conference

Conference	Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004
Country/Territory	Canada
City	Banff, Alta
Period	4/07/04 → 8/07/04

Cite this

@inproceedings{de95bdafd8ba4278babf77c8d19075f9,

title = "Model selection via the AUC",

abstract = "We present a statistical analysis of the AUC as an evaluation criterion for classification scoring models. First, we consider significance tests for the difference between AUC scores of two algorithms on the same test set. We derive exact moments under simplifying assumptions and use them to examine approximate practical methods from the literature. We then compare AUC to empirical misclassification error when the prediction goal is to minimize future error rate. We show that the AUC may be preferable to empirical error even in this case and discuss the tradeoff between approximation error and estimation error underlying this phenomenon.",

author = "Saharon Rosset",

year = "2004",

language = "אנגלית",

isbn = "1581138385",

series = "Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004",

pages = "703--710",

editor = "R. Greiner and D. Schuurmans",

booktitle = "Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004",

note = "Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004 ; Conference date: 04-07-2004 Through 08-07-2004",

}

Rosset, S 2004, Model selection via the AUC. in R Greiner & D Schuurmans (eds), Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004. Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004, pp. 703-710, Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004, Banff, Alta, Canada, 4/07/04.

Model selection via the AUC. / Rosset, Saharon.
Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004. ed. / R. Greiner; D. Schuurmans. 2004. p. 703-710 (Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Model selection via the AUC

AU - Rosset, Saharon

PY - 2004

Y1 - 2004

N2 - We present a statistical analysis of the AUC as an evaluation criterion for classification scoring models. First, we consider significance tests for the difference between AUC scores of two algorithms on the same test set. We derive exact moments under simplifying assumptions and use them to examine approximate practical methods from the literature. We then compare AUC to empirical misclassification error when the prediction goal is to minimize future error rate. We show that the AUC may be preferable to empirical error even in this case and discuss the tradeoff between approximation error and estimation error underlying this phenomenon.

AB - We present a statistical analysis of the AUC as an evaluation criterion for classification scoring models. First, we consider significance tests for the difference between AUC scores of two algorithms on the same test set. We derive exact moments under simplifying assumptions and use them to examine approximate practical methods from the literature. We then compare AUC to empirical misclassification error when the prediction goal is to minimize future error rate. We show that the AUC may be preferable to empirical error even in this case and discuss the tradeoff between approximation error and estimation error underlying this phenomenon.

UR - http://www.scopus.com/inward/record.url?scp=14344255185&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:14344255185

SN - 1581138385

SN - 9781581138382

T3 - Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004

SP - 703

EP - 710

BT - Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004

A2 - Greiner, R.

A2 - Schuurmans, D.

T2 - Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004

Y2 - 4 July 2004 through 8 July 2004

ER -

Model selection via the AUC

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this