TY - GEN
T1 - Lautum regularization for semi-supervised transfer learning
AU - Jakubovitz, Daniel
AU - Rodrigues, Miguel R.D.
AU - Giryes, Raja
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/10
Y1 - 2019/10
N2 - Transfer learning is a very important tool in deep learning as it allows propagating information from one 'source dataset' to another 'target dataset', especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest a novel information theoretic approach for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during the network training on the source dataset. Our theory suggests that one may improve the transferability of a deep neural network by imposing a Lautum information based regularization that relates the network weights to the target data. We demonstrate the effectiveness of the proposed approach in various transfer learning experiments.
AB - Transfer learning is a very important tool in deep learning as it allows propagating information from one 'source dataset' to another 'target dataset', especially in the case of a small number of training examples in the latter. Yet, discrepancies between the underlying distributions of the source and target data are commonplace and are known to have a substantial impact on algorithm performance. In this work we suggest a novel information theoretic approach for the analysis of the performance of deep neural networks in the context of transfer learning. We focus on the task of semi-supervised transfer learning, in which unlabeled samples from the target dataset are available during the network training on the source dataset. Our theory suggests that one may improve the transferability of a deep neural network by imposing a Lautum information based regularization that relates the network weights to the target data. We demonstrate the effectiveness of the proposed approach in various transfer learning experiments.
KW - Information theory
KW - Semi supervised learning
KW - Transfer learning
UR - http://www.scopus.com/inward/record.url?scp=85082438329&partnerID=8YFLogxK
U2 - 10.1109/ICCVW.2019.00100
DO - 10.1109/ICCVW.2019.00100
M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???
AN - SCOPUS:85082438329
T3 - Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019
SP - 763
EP - 767
BT - Proceedings - 2019 International Conference on Computer Vision Workshop, ICCVW 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 17th IEEE/CVF International Conference on Computer Vision Workshop, ICCVW 2019
Y2 - 27 October 2019 through 28 October 2019
ER -