CLUB upper bound on mutual information

Formula: I_CLUB = E[log q(y|x)] - E_marg[log q(y|x)]

Usage

Clbuvc(x, y, q = NULL)

Arguments

x

Observations of X.

y

Observations of Y.

q

Optional (a, b, sigma2) of y | x ~ N(a + b x, sigma2).

Value

List with estimate, club, positive, negative, a, b, sigma2, rho, mi_gauss, n, method.

Details

The second term averages the SAME conditional density over mismatched pairs, so the bound is the average log-ratio between matched and mismatched likelihoods. It is an upper bound on I(X;Y) whenever q equals the true conditional. With a Gaussian q fitted by least squares the value has the closed form b^2 var(x) / sigma2.

References

Cheng, Hao, Dai, Liu, Gan & Carin (2020), CLUB, ICML 119:1779-1788.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
D <- data.frame(x = c(1, 2, 3, 4), y = c(2, 4, 5, 9))
Clbuvc(V, D)
#> $estimate
#> [1] 2.440678
#> 
#> $club
#> [1] 2.440678
#> 
#> $positive
#> [1] -1.647764
#> 
#> $negative
#> [1] -4.088442
#> 
#> $a
#> [1] -0.1071429
#> 
#> $b
#> [1] 0.8571429
#> 
#> $sigma2
#> [1] 1.580357
#> 
#> $rho
#> [1] 0.8422349
#> 
#> $mi_gauss
#> [1] 0.6178343
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "CLUB contrastive log-ratio upper bound on MI"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"