Generalization error of deep neural networks: Role of classification margin and data structure

Jure Sokolić, Raja Giryes, Guillermo Sapiro, Miguel R.D. Rodrigues

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

5 Scopus citations

Abstract

Understanding the generalization properties of deep learning models is critical for their successful usage in many applications, especially in the regimes where the number of training samples is limited. We study the generalization properties of deep neural networks (DNNs) via the Jacobian matrix of the network. Our analysis is general to arbitrary network structures, types of non-linearities and pooling operations. We show that bounding the spectral norm of the Jacobian matrix in the network reduces the generalization error. In addition, we tie this error to the invariance in the data and the network. Experiments on the MNIST and ImageNet datasets support these findings. This short paper summarizes our generalization error theorems for DNNs and for general invariant classifiers [1], [2].

Original languageEnglish
Title of host publication2017 12th International Conference on Sampling Theory and Applications, SampTA 2017
EditorsGholamreza Anbarjafari, Andi Kivinukk, Gert Tamberg
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages147-151
Number of pages5
ISBN (Electronic)9781538615652
DOIs
StatePublished - 1 Sep 2017
Event12th International Conference on Sampling Theory and Applications, SampTA 2017 - Tallinn, Estonia
Duration: 3 Jul 20177 Jul 2017

Publication series

Name2017 12th International Conference on Sampling Theory and Applications, SampTA 2017

Conference

Conference12th International Conference on Sampling Theory and Applications, SampTA 2017
Country/TerritoryEstonia
CityTallinn
Period3/07/177/07/17

Fingerprint

Dive into the research topics of 'Generalization error of deep neural networks: Role of classification margin and data structure'. Together they form a unique fingerprint.

Cite this