Jensen-Shannon divergence between two discrete distributions

Formula: with \(M = (P+Q)/2\), \(JSD(P,Q) = \tfrac12 KL(P\|M) + \tfrac12 KL(Q\|M) = \tfrac12 \sum p \log(2p/(p+q)) + \tfrac12 \sum q \log(2q/(p+q))\). Unlike KL it is symmetric, always finite and bounded by \(\log 2\) – the finiteness is the point of averaging into \(M\), since \(KL(P\|Q)\) is infinite wherever Q puts no mass and P does. Terms with a zero contribute nothing, by \(0\log 0 = 0\). Its square root is a metric.

Usage

Jsdiv(p, q, base = 2, normalize = TRUE)

Arguments

p, q

Non-negative weights over the same support.

base

Logarithm base; 2 gives bits, exp(1) nats.

normalize

Rescale p and q to sum to one first.

Value

List with estimate, distance, bound, base, n, method.

References

Lin (1991), IEEE Transactions on Information Theory 37:145-151. Paywalled at IEEE; the coded form was read from Drost's philentropy, src/distances_internal.h::jensen_shannon_internal (tarball philentropy_0.10.0 from CRAN), with exactly the zero-guards used here.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Jsdiv(V, V)
#> $estimate
#> [1] 0
#> 
#> $distance
#> [1] 0
#> 
#> $bound
#> [1] 1
#> 
#> $base
#> [1] 2
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Jensen-Shannon divergence"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"