TY - JOUR
T1 - Natural type selection in adaptive lossy compression
AU - Zamir, Ram
AU - Rose, Kenneth
N1 - Funding Information:
Manuscript received October 1998; revised November 1999 and July 2000. This work was supported in part by the National Science Foundation under Grant NCR-9314335, the Binational Science Foundation 9800309, the University of California MICRO Program, Cisco Systems, Inc., Conexant Systems, Inc., Dialogic Corp., Fujitsu Laboratories of America, Inc., General Electric Co., Hughes Network Systems, Lernout & Hauspie Speech Products, Lockheed Martin, Lucent Technologies, Inc., Qualcomm, Inc., and Texas Instruments, Inc. The material in this paper was presented in part at the Information Theory Workshop, Haifa, Israel, June 1996, at the International Symposium on Information Theory, Ulm, Germany, June 1997, and at the Canadian Workshop on Information Theory, Kingston, ON, Canada, June 1999.
PY - 2001/1
Y1 - 2001/1
N2 - Consider approximate (lossy) matching of a source string ∼P, with a random codebook generated from reproduction distribution Q, at a specified distortion d. Recent work determined the minimum coding rate R 1 = R(P, Q, d) for this setting. We observe that for large word length and with high probability, the matching codeword is typical with a distribution Q 1 which is different from Q. If a new random codebook is generated ∼Q 1, then the source string will favor codewords which are typical with a new distribution Q 2, resulting in minimum coding rate R 2 = R(P, Q 1, d), and so on. We show that the sequences of distributions Q 1, Q 2, ... and rates R 1, R 2, ..., generated by this procedure, converge to an optimum reproduction distribution Q*, and the rate-distortion function R(P, d), respectively. We also derive a fixed rate-distortion slope version of this natural type selection process. In the latter case, an iteration of the process stochastically simulates an iteration of the Blahut-Arimoto (BA) algorithm for rate-distortion function computation (without recourse to prior knowledge of the underlying source distribution). To strengthen these limit statements, we also characterize the steady-state error of these procedures when iterating at a finite string length. Implications of the main results provide fresh insights into the workings of lossy variants of the Lempel-Ziv algorithm for adaptive compression.
AB - Consider approximate (lossy) matching of a source string ∼P, with a random codebook generated from reproduction distribution Q, at a specified distortion d. Recent work determined the minimum coding rate R 1 = R(P, Q, d) for this setting. We observe that for large word length and with high probability, the matching codeword is typical with a distribution Q 1 which is different from Q. If a new random codebook is generated ∼Q 1, then the source string will favor codewords which are typical with a new distribution Q 2, resulting in minimum coding rate R 2 = R(P, Q 1, d), and so on. We show that the sequences of distributions Q 1, Q 2, ... and rates R 1, R 2, ..., generated by this procedure, converge to an optimum reproduction distribution Q*, and the rate-distortion function R(P, d), respectively. We also derive a fixed rate-distortion slope version of this natural type selection process. In the latter case, an iteration of the process stochastically simulates an iteration of the Blahut-Arimoto (BA) algorithm for rate-distortion function computation (without recourse to prior knowledge of the underlying source distribution). To strengthen these limit statements, we also characterize the steady-state error of these procedures when iterating at a finite string length. Implications of the main results provide fresh insights into the workings of lossy variants of the Lempel-Ziv algorithm for adaptive compression.
KW - Adaptive compression
KW - Alternating optimization
KW - Approximate string matching
KW - Blahut-Arimoto algorithm
KW - Lempel-Ziv coding
KW - Rate-distortion
KW - Typical sequences
KW - Universal coding
UR - http://www.scopus.com/inward/record.url?scp=0035089002&partnerID=8YFLogxK
U2 - 10.1109/18.904515
DO - 10.1109/18.904515
M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???
AN - SCOPUS:0035089002
SN - 0018-9448
VL - 47
SP - 99
EP - 111
JO - IEEE Transactions on Information Theory
JF - IEEE Transactions on Information Theory
IS - 1
ER -