Chain rule for relative entropy

$$D(p(x,y)\|q(x,y)) = D(p(x)\|q(x)) + D(p(y|x)\|q(y|x))$$ Cover & Thomas (2006), Theorem 2.5.3, eq. (2.67), p. 24, with the conditional relative entropy of eq. (2.65).

Usage

Klchain(pxy, qxy, base = 2)

Arguments

pxy, qxy

Joint pmfs of the same shape; each normalised here.

base

Log base; 2 gives bits.

Value

Named list with estimate, marginal, conditional, residual, n, method. residual is zero up to rounding.

References

Cover & Thomas (2006), eq. (2.65), (2.67).

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Klchain(V, V)
#> $estimate
#> [1] 0
#> 
#> $marginal
#> [1] 0
#> 
#> $conditional
#> [1] 0
#> 
#> $residual
#> [1] 0
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Chain rule for relative entropy -- Cover & Thomas (2006) eq. (2.67)"
#>