Expected code length of a predictive model (the wrong-code bound)

$$H(p) + D(p\|q) \le E_p\,l(X) < H(p) + D(p\|q) + 1$$ Cover & Thomas (2006), Theorem 5.4.3, eq. (5.42), p. 115.

Usage

Predcomp(model, data, base = 2)

Arguments

model

Predictive pmf over an alphabet of size K; normalised here.

data

Observed symbols as 0-based indices into that alphabet (0-based, matching the Python arm).

base

Log base; 2 gives bits per symbol.

Value

Named list with estimate, entropy, kl, upper, n, method.

References

Cover & Thomas (2006), Theorem 5.4.3.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
M <- matrix(c(1, 2, 3, 4, 5, 6), nrow = 2)
Predcomp(V, M)
#> $estimate
#> [1] 3.120057
#> 
#> $entropy
#> [1] 2.584963
#> 
#> $kl
#> [1] 0.5350945
#> 
#> $upper
#> [1] 4.120057
#> 
#> $n
#> [1] 6
#> 
#> $method
#> [1] "Wrong-code expected length H(p)+D(p||q) -- Cover & Thomas (2006) Thm 5.4.3"
#>