SOURCE. Barber, D. and Agakov, F. (2003), "The IM Algorithm: A Variational Approach to Information Maximization", Advances in Neural Information Processing Systems 16 (NIPS 2003), MIT Press.
Value
List with bound, entropy_x, entropy_y,
expected_log_q, plugin_mi, gap,
conditional_entropy, n, nx, ny.
Details
Write I(X;Y) = H(X) - H(X|Y) and replace the intractable posterior
p(x|y) by a variational decoder q(x|y). Since H(X|Y) =
-E[log q(x|y)] - E_{p(y)}[KL(p(.|y) || q(.|y))] and the KL is
non-negative, dropping it can only lower the value:
I(X;Y) >= H(X) + E_{p(x,y)}[log q(x|y)], with equality exactly when
q(x|y) = p(x|y) for every y that occurs. The gap is the average KL, so
gap is non-negative by Gibbs' inequality – a property this
module asserts rather than assumes.
SCOPE. Discrete X and Y with the joint estimated by counting; all
quantities in nats. q is an ny-by-nx matrix whose row j is
q(. | y = level j); omitted, the empirical conditional is used, making
the bound tight and equal to the plug-in mutual information. A
continuous version would need a parametric decoder and an optimiser
and is not implemented – this implementation's scope choice.
The form I(X;Y) >= E[log q(y|x)/p(y)] is the same bound with X and Y exchanged; call the function with the arguments swapped.