TY - JOUR
T1 - Scale-sensitive dimensions, uniform convergence, and learnability
AU - Alon, Noga
AU - Ben-David, Shai
AU - Cesa-Bianchi, Nicolò
AU - Haussler, David
PY - 1997/7
Y1 - 1997/7
N2 - Learnability in Valiant's PAC learning model has been shown to be strongly related to the existence of uniform laws of large numbers. These laws define a distribution-free convergence property of means to expectations uniformly over classes of random variables. Classes of real-valued functions enjoying such a property are also known as uniform Glivenko-Cantelli classes. In this paper, we prove, through a generalization of Sauer's lemma that may be interesting in its own right, a new characterization of uniform Glivenko-Cantelli classes. Our characterization yields Dudley, Giné and Zinn's previous characterization as a corollary. Furthermore, it is the first based on a simple combinatorial quantity generalizing the Vapnik-Chervonenkis dimension. We apply this result to obtain the weakest combinatorial condition known to imply PAC learnability in the statistical regression (or "agnostic") framework. Furthermore, we find a characterization of learnability in the probabilistic concept model, solving an open problem posed by Kearns and Schapire. These results show that the accuracy parameter plays a crucial role in determining the effective complexity of the learner's hypothesis class.
AB - Learnability in Valiant's PAC learning model has been shown to be strongly related to the existence of uniform laws of large numbers. These laws define a distribution-free convergence property of means to expectations uniformly over classes of random variables. Classes of real-valued functions enjoying such a property are also known as uniform Glivenko-Cantelli classes. In this paper, we prove, through a generalization of Sauer's lemma that may be interesting in its own right, a new characterization of uniform Glivenko-Cantelli classes. Our characterization yields Dudley, Giné and Zinn's previous characterization as a corollary. Furthermore, it is the first based on a simple combinatorial quantity generalizing the Vapnik-Chervonenkis dimension. We apply this result to obtain the weakest combinatorial condition known to imply PAC learnability in the statistical regression (or "agnostic") framework. Furthermore, we find a characterization of learnability in the probabilistic concept model, solving an open problem posed by Kearns and Schapire. These results show that the accuracy parameter plays a crucial role in determining the effective complexity of the learner's hypothesis class.
KW - PAC learning
KW - Theory
KW - Uniform laws of large numbers
KW - Vapnik-Chervonenkis dimension
UR - http://www.scopus.com/inward/record.url?scp=0031176507&partnerID=8YFLogxK
U2 - 10.1145/263867.263927
DO - 10.1145/263867.263927
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
AN - SCOPUS:0031176507
SN - 0004-5411
VL - 44
SP - 615
EP - 631
JO - Journal of the ACM
JF - Journal of the ACM
IS - 4
ER -