Ranking-based evaluation of regression models

Saharon Rosset, Claudia Perlich, Bianca Zadrozny

Research output: Contribution to journalArticlepeer-review

Abstract

We suggest the use of ranking-based evaluation measures for regression models, as a complement to the commonly used residual-based evaluation. We argue that in some cases, such as the case study we present, ranking can be the main underlying goal in building a regression model, and ranking performance is the correct evaluation metric. However, even when ranking is not the contextually correct performance metric, the measures we explore still have significant advantages: They are robust against extreme outliers in the evaluation set; and they are interpretable. The two measures we consider correspond closely to non-parametric correlation coefficients commonly used in data analysis (Spearman's ρ and Kendall's τ); and they both have interesting graphical representations, which, similarly to ROC curves, offer useful various model performance views, in addition to a one-number summary in the area under the curve. An interesting extension which we explore is to evaluate models on their performance in "partially" ranking the data, which we argue can better represent the utility of the model in many cases. We illustrate our methods on a case study of evaluating IT Wallet size estimation models for IBM's customers.

Original languageEnglish
Pages (from-to)331-353
Number of pages23
JournalKnowledge and Information Systems
Volume12
Issue number3
DOIs
StatePublished - Aug 2007
Externally publishedYes

Keywords

  • Evaluation robustness
  • Model evaluation
  • Performance visualization
  • Ranking correlation
  • Regression

Cite this