Restricted Boltzmann machine (contrastive divergence)

"Restricted" means no within-layer edges, so the conditionals factorise and block Gibbs sampling is cheap. The partition function is a sum over all 2^(|v| + |h|) configurations and is intractable, so the likelihood gradient cannot be computed; CD-k substitutes a k-step chain started at the data.

Usage

morie_esl_boltzmann(
  v,
  h = 4L,
  lr = 0.1,
  n_epochs = 200L,
  k_cd = 1L,
  seed = 0L,
  batch_size = NULL
)

Arguments

v

Binary visible data, n by d, entries in 0/1.

h

Number of hidden units.

lr

Learning rate.

n_epochs

Training epochs.

k_cd

Gibbs steps per update.

seed

Seed.

batch_size

Mini-batch size; defaults to the full data.

Value

List with W, a, b, hidden_prob, reconstruction, reconstruction_error, error_path, free_energy.

Details

That makes this a biased gradient of an approximate objective, not the likelihood – so NO log-likelihood is reported. reconstruction_error is reported instead, with the caveat that it can fall while the model gets worse, which is why RBM training is judged by samples.

References

Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation 14(8), 1771-1800.

Examples

base <- rbind(c(1, 1, 1, 0, 0, 0), c(0, 0, 0, 1, 1, 1))
V <- base[rep(1:2, each = 50), ]
r <- morie_esl_boltzmann(V, h = 3, lr = 0.5, n_epochs = 200L, seed = 1L)
r$error_path[length(r$error_path)] < r$error_path[1]
#> [1] TRUE