Universal finite memory machines for coding binary sequences

In this work we consider the problem of universal sequential probability assignment, under the self-information loss, where the machine for performing the universal probability assignment is constrained to have a finite memory. Sequential probability assignment is equivalent to lossless source coding if we ignore the number of states required to convert the probability estimate into code bits. We consider both the probabilistic setting where the sequence is generated by a probabilistic source (either Bernoulli i.i.d. source or q-th order Markov source), and the deterministic setting where the sequence is a deterministic individual sequence. We also consider the case where the universal machine is deterministic, randomized, time-invariant or time-variant. We provide in most cases lower bounds and describe finite memory universal machines whose performance, in terms of the memory size, is compared to these bounds.