Information bottleneck for Gaussian variables

Gal Chechik, Amir Globerson, Naftali Tishby, Yair Weiss

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The problem of extracting the relevant aspects of data was addressed through the information bottleneck (IB) method, by (soft) clustering one variable while preserving information about another - relevance - variable. An interesting question addressed in the current work is the extension of these ideas to obtain continuous representations that preserve relevant information, rather than discrete clusters. We give a formal definition of the general continuous IB problem and obtain an analytic solution for the optimal representation for the important case of multivariate Gaussian variables. The obtained optimal representation is a noisy linear projection to eigenvectors of the normalized correlation matrix ∑x|y-1 x , which is also the basis obtained in Canonical Correlation Analysis. However, in Gaussian IB, the compression tradeoff parameter uniquely determines the dimension, as well as the scale of each eigenvector. This introduces a novel interpretation where solutions of different ranks lie on a continuum parametrized by the compression level. Our analysis also provides an analytic expression for the optimal tradeoff - The information curve - in terms of the eigenvalue spectrum.

Original languageEnglish
Title of host publicationAdvances in Neural Information Processing Systems 16 - Proceedings of the 2003 Conference, NIPS 2003
PublisherNeural information processing systems foundation
ISBN (Print)0262201526, 9780262201520
StatePublished - 2004
Externally publishedYes
Event17th Annual Conference on Neural Information Processing Systems, NIPS 2003 - Vancouver, BC, Canada
Duration: 8 Dec 200313 Dec 2003

Publication series

NameAdvances in Neural Information Processing Systems
ISSN (Print)1049-5258

Conference

Conference17th Annual Conference on Neural Information Processing Systems, NIPS 2003
Country/TerritoryCanada
CityVancouver, BC
Period8/12/0313/12/03

Fingerprint

Dive into the research topics of 'Information bottleneck for Gaussian variables'. Together they form a unique fingerprint.

Cite this