A theoretical framework for deep transfer learning

Tomer Galanti, Lior Wolf*, Tamir Hazan

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

21 Scopus citations

Abstract

We generalize the notion of PAC learning to include transfer learning. In our framework, the linkage between the source and the target tasks is a result of having the sample distribution of all classes drawn from the same distribution of distributions, and by restricting all source and a target concepts to belong to the same hypothesis subclass. We have two models: An adversary model and a randomized model. In the adversary model, we show that for binary classification, conventional PAC-learning is equivalent to the new notion of PAC-transfer and to transfer generalization of the VC-dimension. For regression, we show that PAC-transferability may exist even in the absence of PAC-learning. In both adversary and randomized models, we provide PAC-Bayesian and VC-style generalization bounds to transfer learning. In the randomized model, we provide bounds specifically derived for Deep Learning. A wide discussion on the tradeoffs between the different involved parameters in the bounds is provided. We demonstrate both cases in which transfer does not reduce the sample size ('trivial transfer') and cases in which the sample size is reduced ('non-trivial transfer').

Original languageEnglish
Pages (from-to)159-209
Number of pages51
JournalInformation and Inference
Volume5
Issue number2
DOIs
StatePublished - Jun 2016

Funding

FundersFunder number
German-Israeli Foundation for Scientific Research and Development

    Keywords

    • Deep learning
    • PAC learning
    • PAC-Bayesian
    • Transfer learning

    Fingerprint

    Dive into the research topics of 'A theoretical framework for deep transfer learning'. Together they form a unique fingerprint.

    Cite this