On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

Jeremias Sulam, Aviad Aberdam, Amir Beck, Michael Elad

Research output: Contribution to journalArticlepeer-review

Abstract

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting from unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.

Original languageEnglish
Article number8664165
Pages (from-to)1968-1980
Number of pages13
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume42
Issue number8
DOIs
StatePublished - 1 Aug 2020

Keywords

  • Multi-layer convolutional sparse coding
  • iterative shrinkage algorithms
  • network unfolding
  • recurrent neural networks

Fingerprint

Dive into the research topics of 'On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks'. Together they form a unique fingerprint.

Cite this