Denoising score-matching objective

Averages \(\lVert s_\theta(\tilde x) + (\tilde x - x)/\sigma^2 \rVert^2\) over Gaussian perturbations.

Usage

morie_diffusion_score_matching(x, score, sigma = 0.1, n_noise = 16, seed = 0)

Arguments

x

data matrix.

score

function mapping a matrix to the score at those points.

sigma

noise scale.

n_noise

perturbations averaged per point.

seed

integer seed.

Value

list with objective, per_sample, target_norm.

Details

Vincent's identity is what makes this tractable: the explicit score-matching objective contains the trace of the score Jacobian, which costs a backward pass per dimension, and the denoising form removes it entirely by giving each noisy sample a known target.

The objective targets the NOISED distribution \(p_\sigma\), not p. A small sigma reduces that bias but leaves the score unconstrained wherever data is sparse, which is why practical use anneals sigma rather than picking one.

References

Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural Computation, 23(7), 1661-1674.

Examples

set.seed(1)
X <- matrix(rnorm(60), ncol = 2)
r <- morie_diffusion_score_matching(X, function(z) -z, sigma = 0.5)
r$objective > 0
#> [1] TRUE