TY - JOUR

T1 - Improved Generalization Bounds for Adversarially Robust Learning

AU - Attias, Idan

AU - Kontorovich, Aryeh

AU - Mansour, Yishay

N1 - Publisher Copyright:
©2022 Idan Attias, Aryeh Kontorovich, and Yishay Mansour.

PY - 2022/6/1

Y1 - 2022/6/1

N2 - We consider a model of robust learning in an adversarial environment. The learner gets uncorrupted training data with access to possible corruptions that may be affected by the adversary during testing. The learner’s goal is to build a robust classifier, which will be tested on future adversarial examples. The adversary is limited to k possible corruptions for each input. We model the learner-adversary interaction as a zero-sum game. This model is closely related to the adversarial examples model of Schmidt et al. (2018); Madry et al. (2017). Our main results consist of generalization bounds for the binary and multiclass classification, as well as the real-valued case (regression). For the binary classification setting, we both tighten the generalization bound of Feige et al. (2015), and are also able to handle infinite hypothesis classes. The sample complexity is improved from O( 14 log(|H|δ )) to O( 12 (k VC(H) log 3 2 +α(k VC(H)) + log(1δ )) for any α > 0. Additionally, we extend the algorithm and generalization bound from the binary to the multiclass and real-valued cases. Along the way, we obtain results on fat-shattering dimension and Rademacher complexity of k-fold maxima over function classes; these may be of independent interest. For binary classification, the algorithm of Feige et al. (2015) uses a regret minimization algorithm and an ERM oracle as a black box; we adapt it for the multiclass and regression settings. The algorithm provides us with near-optimal policies for the players on a given training sample.

AB - We consider a model of robust learning in an adversarial environment. The learner gets uncorrupted training data with access to possible corruptions that may be affected by the adversary during testing. The learner’s goal is to build a robust classifier, which will be tested on future adversarial examples. The adversary is limited to k possible corruptions for each input. We model the learner-adversary interaction as a zero-sum game. This model is closely related to the adversarial examples model of Schmidt et al. (2018); Madry et al. (2017). Our main results consist of generalization bounds for the binary and multiclass classification, as well as the real-valued case (regression). For the binary classification setting, we both tighten the generalization bound of Feige et al. (2015), and are also able to handle infinite hypothesis classes. The sample complexity is improved from O( 14 log(|H|δ )) to O( 12 (k VC(H) log 3 2 +α(k VC(H)) + log(1δ )) for any α > 0. Additionally, we extend the algorithm and generalization bound from the binary to the multiclass and real-valued cases. Along the way, we obtain results on fat-shattering dimension and Rademacher complexity of k-fold maxima over function classes; these may be of independent interest. For binary classification, the algorithm of Feige et al. (2015) uses a regret minimization algorithm and an ERM oracle as a black box; we adapt it for the multiclass and regression settings. The algorithm provides us with near-optimal policies for the players on a given training sample.

KW - Adversarial Robustness

KW - PAC Learning

KW - Sample Complexity

KW - Zero-Sum Game

UR - http://www.scopus.com/inward/record.url?scp=85146941964&partnerID=8YFLogxK

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:85146941964

SN - 1532-4435

VL - 23

JO - Journal of Machine Learning Research

JF - Journal of Machine Learning Research

ER -