SOURCE. Jordan, M.I., Ghahramani, Z., Jaakkola, T.S. and Saul, L.K. (1999), "An Introduction to Variational Methods for Graphical Models", Machine Learning 37(2):183-233, doi:10.1023/A:1007665907178.
Usage
Vinfer(
log_p = "normal-gamma",
q_family = "meanfield",
x = NULL,
mu0 = 0,
lambda0 = 0,
a0 = 0,
b0 = 0,
max_iter = 200,
tol = 1e-12
)Value
List with mu_n, lambda_n, a_n, b_n,
e_tau, e_mu, var_mu, elbo,
elbo_path, elbo_monotone, iterations,
converged, n.
Details
The mean-field family factorises q(z) = prod_j q_j(z_j), and Section 3
gives the coordinate update that maximises the lower bound with the
other factors fixed: log q\*_j(z_j) = E_{q_{-j}}[log p(x, z)] + const.
The bound is ELBO(q) = E_q[log p(x,z)] - E_q[log q(z)] <= log p(x), and
no coordinate update may decrease it – asserted here as
elbo_monotone rather than assumed.
MODEL. The general update is not executable without a joint, so the joint is the conjugate univariate Normal-Gamma: x_n | mu, tau ~ N(mu, 1/tau), mu | tau ~ N(mu0, 1/(lambda0 tau)), tau ~ Gamma(a0, b0), with q(mu, tau) = q(mu) q(tau). The updates are closed form: mu_N = (lambda0 mu0 + N xbar)/(lambda0 + N), lambda_N = (lambda0 + N) E[tau], a_N = a0 + (N+1)/2, b_N = b0 + E_mu[sum (x_n - mu)^2 + lambda0 (mu - mu0)^2]/2, E[tau] = a_N/b_N.
Only this joint is implemented; that is this implementation's scope choice, stated rather than attributed – a general log_p callable cannot cross the Python/R boundary and would make the arms untestable.
ANCHOR. In the improper limit lambda0 = a0 = b0 = 0, mu_N = xbar exactly and the fixed point solves t = (N+1)/(S + 1/t) with S = sum (x_n - xbar)^2, giving t = N/S, the maximum likelihood precision.
References
Jordan, M.I., Ghahramani, Z., Jaakkola, T.S. and Saul, L.K. (1999). Machine Learning 37(2):183-233. doi:10.1023/A:1007665907178.
Examples
Vinfer("normal-gamma", "meanfield", c(1, 2, 3, 4, 5))$e_tau
#> [1] 0.5