Batch-normalisation running statistics and inference transform

runmean <- (1-momentum) runmean + momentum mean(x); runvar <- (1-momentum) runvar + momentum (m/(m-1)) biased_var(x); y = gamma (x - runmean)/sqrt(runvar + eps) + beta.

Usage

Bnrunstat(
  x,
  runmean = 0,
  runvar = 1,
  momentum = 0.1,
  eps = 1e-05,
  gamma = 1,
  beta = 0
)

Arguments

x

Mini-batch of activations, length m >= 2.

runmean, runvar

Running population moments.

momentum

Weight given to the new batch, in [0, 1].

eps

Numerical floor inside the square root.

gamma, beta

Learned scale and shift.

Value

List with runmean, runvar, batchmean, batchvar, batchvarunb, normalized, trainnorm, m.

References

Ioffe and Szegedy (2015), arXiv:1502.03167, Sect. 3.1 and Algorithm 2. Read from the ar5iv rendering; the same paper is in the local corpus.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Bnrunstat(V)
#> $runmean
#> [1] 0.45
#> 
#> $runvar
#> [1] 1.5
#> 
#> $batchmean
#> [1] 4.5
#> 
#> $batchvar
#> [1] 5.25
#> 
#> $batchvarunb
#> [1] 6
#> 
#> $normalized
#> [1] 0.4490716 1.2655655 2.0820593 2.8985532 3.7150471 4.5315409 5.3480348
#> [8] 6.1645286
#> 
#> $trainnorm
#> [1] -1.5275238 -1.0910884 -0.6546530 -0.2182177  0.2182177  0.6546530  1.0910884
#> [8]  1.5275238
#> 
#> $m
#> [1] 8
#> 
#> $method
#> [1] "Batch-norm running statistics (Ioffe-Szegedy 2015 Sect. 3.1)"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"