TY - JOUR

T1 - Hierarchical universal coding

AU - Feder, Meir

AU - Merhav, Neri

N1 - Funding Information:
Manuscript received April 27, 1994; revised January 20, 1996. This work was supported in part by the S. Neaman Institute and the Wolfson Research Awards administered by the Israel Academy of Science and Humanities. M. Feder is with the Department of Electrical Engineering-Systems, Tel Aviv University, Tel Aviv 69978, Israel. N. Merhav is with the Department of Electrical Engineering, Tech-nion-Israel Institute of Technology, Haifa 32000, Israel. Publisher Item Identifier S 0018-9448(96)0545 1-X.

PY - 1996

Y1 - 1996

N2 - In an earlier paper, we proved a strong version of the redundancy-capacity converse theorem of universal coding, stating that for "most" sources in a given class, the universal coding redundancy is essentially lower-bounded by the capacity of the channel induced by this class. Since this result holds for general classes of sources, it extends Rissanen's strong converse theorem for parametric families. While our earlier result has established strong optimality only for mixture codes weighted by the capacity-achieving prior, our first result herein extends this finding to a general prior. For some cases our technique also leads to a simplified proof of the above mentioned strong converse theorem. The major interest in this paper, however, is in extending the theory of universal coding to hierarchical structures of classes, where each class may have a different capacity. In this setting, one wishes to incur redundancy essentially as small as that corresponding to the active class, and not the union of classes. Our main result is that the redundancy of a code based on a two-stage mixture (first, within each class, and then over the classes), is no worse than that of any other code for "most" sources of "most" classes. If, in addition, the classes can be efficiently distinguished by a certain decision rule, then the best attainable redundancy is given explicitly by the capacity of the active class plus the normalized negative logarithm of the prior probability assigned to this class. These results suggest some interesting guidelines as for the choice of the prior. We also discuss some examples with a natural hierarchical partition into classes.

AB - In an earlier paper, we proved a strong version of the redundancy-capacity converse theorem of universal coding, stating that for "most" sources in a given class, the universal coding redundancy is essentially lower-bounded by the capacity of the channel induced by this class. Since this result holds for general classes of sources, it extends Rissanen's strong converse theorem for parametric families. While our earlier result has established strong optimality only for mixture codes weighted by the capacity-achieving prior, our first result herein extends this finding to a general prior. For some cases our technique also leads to a simplified proof of the above mentioned strong converse theorem. The major interest in this paper, however, is in extending the theory of universal coding to hierarchical structures of classes, where each class may have a different capacity. In this setting, one wishes to incur redundancy essentially as small as that corresponding to the active class, and not the union of classes. Our main result is that the redundancy of a code based on a two-stage mixture (first, within each class, and then over the classes), is no worse than that of any other code for "most" sources of "most" classes. If, in addition, the classes can be efficiently distinguished by a certain decision rule, then the best attainable redundancy is given explicitly by the capacity of the active class plus the normalized negative logarithm of the prior probability assigned to this class. These results suggest some interesting guidelines as for the choice of the prior. We also discuss some examples with a natural hierarchical partition into classes.

KW - Arbitrarily varying sources

KW - Capacity

KW - Inaximin redundancy

KW - Minimax redundancy

KW - Mixtures

KW - Redundancy-capacity theorem

KW - Universal coding

UR - http://www.scopus.com/inward/record.url?scp=0030241059&partnerID=8YFLogxK

U2 - 10.1109/18.532877

DO - 10.1109/18.532877

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:0030241059

SN - 0018-9448

VL - 42

SP - 1354

EP - 1364

JO - IEEE Transactions on Information Theory

JF - IEEE Transactions on Information Theory

IS - 5

ER -