Finite-memory universal prediction of individual sequences

Eado Meron*, Meir Feder

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

16 Scopus citations


The problem of predicting the next outcome of an individual binary sequence under the constraint that the universal predictor has a finite memory, is explored. In this analysis, the finite-memory universal predictors are either deterministic or random time-invariant finite-state (FS) machines with K states (K-state machines). The paper provides bounds on the asymptotic achievable regret of these constrained universal predictors as a function of K, the number of their states, for long enough sequences. The specific results are as follows. When the universal predictors are deterministic machines, the comparison class consists of constant predictors, and prediction is with respect to the 0-1 loss function (Hamming distance), we get tight bounds indicating that the optimal asymptotic regret is 1/(2K). In that case of K-state deterministic universal predictors, the constant predictors comparison class, but prediction is with respect to the self-information (code length) and the square-error loss functions, we show an upper bound on the regret (coding redundancy) of O(K-2/3) and a lower bound of θ(K-4/5). For these loss functions, if the predictor is allowed to be a random K-state machine, i.e., a machine with random state transitions, we get a lower bound of θ (1/K) on the regret, with a matching upper bound of O (1/K) for the square-error loss, and an upper bound of O (log K/K)1 for the self-information loss. In addition, we provide results for all these loss functions in the case where the comparison class consists of all predictors that are order-L Markov machines.

Original languageEnglish
Pages (from-to)1506-1523
Number of pages18
JournalIEEE Transactions on Information Theory
Issue number7
StatePublished - Jul 2004


FundersFunder number
Intel Israel
Yitzhak and Chaya Weinstein Institute for Research in Signal


    • Exponentially decaying memory
    • FS prediction
    • Finite-state (FS) machines
    • Imaginary sliding window
    • Saturated counter (SC)
    • Universal coding
    • Universal prediction


    Dive into the research topics of 'Finite-memory universal prediction of individual sequences'. Together they form a unique fingerprint.

    Cite this