TY - JOUR

T1 - Natural type selection in adaptive lossy compression

AU - Zamir, Ram

AU - Rose, Kenneth

N1 - Funding Information:
Manuscript received October 1998; revised November 1999 and July 2000. This work was supported in part by the National Science Foundation under Grant NCR-9314335, the Binational Science Foundation 9800309, the University of California MICRO Program, Cisco Systems, Inc., Conexant Systems, Inc., Dialogic Corp., Fujitsu Laboratories of America, Inc., General Electric Co., Hughes Network Systems, Lernout & Hauspie Speech Products, Lockheed Martin, Lucent Technologies, Inc., Qualcomm, Inc., and Texas Instruments, Inc. The material in this paper was presented in part at the Information Theory Workshop, Haifa, Israel, June 1996, at the International Symposium on Information Theory, Ulm, Germany, June 1997, and at the Canadian Workshop on Information Theory, Kingston, ON, Canada, June 1999.

PY - 2001/1

Y1 - 2001/1

N2 - Consider approximate (lossy) matching of a source string ∼P, with a random codebook generated from reproduction distribution Q, at a specified distortion d. Recent work determined the minimum coding rate R 1 = R(P, Q, d) for this setting. We observe that for large word length and with high probability, the matching codeword is typical with a distribution Q 1 which is different from Q. If a new random codebook is generated ∼Q 1, then the source string will favor codewords which are typical with a new distribution Q 2, resulting in minimum coding rate R 2 = R(P, Q 1, d), and so on. We show that the sequences of distributions Q 1, Q 2, ... and rates R 1, R 2, ..., generated by this procedure, converge to an optimum reproduction distribution Q*, and the rate-distortion function R(P, d), respectively. We also derive a fixed rate-distortion slope version of this natural type selection process. In the latter case, an iteration of the process stochastically simulates an iteration of the Blahut-Arimoto (BA) algorithm for rate-distortion function computation (without recourse to prior knowledge of the underlying source distribution). To strengthen these limit statements, we also characterize the steady-state error of these procedures when iterating at a finite string length. Implications of the main results provide fresh insights into the workings of lossy variants of the Lempel-Ziv algorithm for adaptive compression.

AB - Consider approximate (lossy) matching of a source string ∼P, with a random codebook generated from reproduction distribution Q, at a specified distortion d. Recent work determined the minimum coding rate R 1 = R(P, Q, d) for this setting. We observe that for large word length and with high probability, the matching codeword is typical with a distribution Q 1 which is different from Q. If a new random codebook is generated ∼Q 1, then the source string will favor codewords which are typical with a new distribution Q 2, resulting in minimum coding rate R 2 = R(P, Q 1, d), and so on. We show that the sequences of distributions Q 1, Q 2, ... and rates R 1, R 2, ..., generated by this procedure, converge to an optimum reproduction distribution Q*, and the rate-distortion function R(P, d), respectively. We also derive a fixed rate-distortion slope version of this natural type selection process. In the latter case, an iteration of the process stochastically simulates an iteration of the Blahut-Arimoto (BA) algorithm for rate-distortion function computation (without recourse to prior knowledge of the underlying source distribution). To strengthen these limit statements, we also characterize the steady-state error of these procedures when iterating at a finite string length. Implications of the main results provide fresh insights into the workings of lossy variants of the Lempel-Ziv algorithm for adaptive compression.

KW - Adaptive compression

KW - Alternating optimization

KW - Approximate string matching

KW - Blahut-Arimoto algorithm

KW - Lempel-Ziv coding

KW - Rate-distortion

KW - Typical sequences

KW - Universal coding

UR - http://www.scopus.com/inward/record.url?scp=0035089002&partnerID=8YFLogxK

U2 - 10.1109/18.904515

DO - 10.1109/18.904515

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:0035089002

SN - 0018-9448

VL - 47

SP - 99

EP - 111

JO - IEEE Transactions on Information Theory

JF - IEEE Transactions on Information Theory

IS - 1

ER -