Kullback-Leibler divergence with additive smoothing for sparse counts

Additive (Lidstone) smoothing replaces a maximum-likelihood count ratio by (c + eps)/(N + V eps), with V the vocabulary size. Without it the Kullback-Leibler divergence of two sparse count vectors is infinite the moment the second assigns zero to an event the first does not, which is the failure mode the smoothing exists to prevent. Both vectors are smoothed with the same eps, so the divergence is finite by construction. The reverse divergence and their sum (Kullback's symmetric divergence J) are reported too, since KL is not symmetric.

Usage

Klmsm1(p, q, eps)

Arguments

p

Non-negative counts over the vocabulary.

q

Non-negative counts over the same vocabulary.

eps

Additive smoothing constant, strictly positive.

Value

List with estimate, kl_pq, kl_qp, symmetric_kl, eps, vocabulary, zeros_p, zeros_q, mass_p, mass_q, n, method.

Details

Formula: P = (c + eps)/(N + V eps); KL(P||Q) = sum P log(P/Q).

References

Chen and Goodman (1996), An empirical study of smoothing techniques for language modeling, Proceedings of the 34th Annual Meeting of the ACL, pp. 310-318, section 2.1. doi:10.3115/981863.981904

Examples

Klmsm1(p = 0.5, q = 0.5, eps = 0.5)
#> $estimate
#> [1] 0
#> 
#> $kl_pq
#> [1] 0
#> 
#> $kl_qp
#> [1] 0
#> 
#> $symmetric_kl
#> [1] 0
#> 
#> $eps
#> [1] 0.5
#> 
#> $vocabulary
#> [1] 1
#> 
#> $zeros_p
#> [1] 0
#> 
#> $zeros_q
#> [1] 0
#> 
#> $mass_p
#> [1] 0.5
#> 
#> $mass_q
#> [1] 0.5
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "P = (c + eps)/(N + V eps) then KL(P||Q) = sum P log(P/Q), Chen & Goodman (1996)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"