Batch normalization with running statistics (Geron Ch 11, morie.fn hmbntr)

Batch statistics use the POPULATION variance; the INFERENCE running variance uses the unbiased (n-1) batch estimate, per the original paper. When running_mean/running_var are absent the batch values become the running values outright.

Usage

morie_geron_batch_normalization(
  x,
  gamma = 1,
  beta = 0,
  eps = 1e-05,
  momentum = 0.9,
  running_mean = NULL,
  running_var = NULL
)

Arguments

x

Mini-batch (n, d) or vector.

gamma, beta

Scale and shift.

eps

Variance floor.

momentum

EMA factor in [0, 1].

running_mean, running_var

Optional previous statistics.

Value

List with y, x_hat, mu, var, running_mean, running_var.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_batch_normalization(V)
#> $y
#>            [,1]
#> [1,] -1.5275238
#> [2,] -1.0910884
#> [3,] -0.6546530
#> [4,] -0.2182177
#> [5,]  0.2182177
#> [6,]  0.6546530
#> [7,]  1.0910884
#> [8,]  1.5275238
#> 
#> $x_hat
#>            [,1]
#> [1,] -1.5275238
#> [2,] -1.0910884
#> [3,] -0.6546530
#> [4,] -0.2182177
#> [5,]  0.2182177
#> [6,]  0.6546530
#> [7,]  1.0910884
#> [8,]  1.5275238
#> 
#> $mu
#> [1] 4.5
#> 
#> $var
#> [1] 5.25
#> 
#> $running_mean
#> [1] 4.5
#> 
#> $running_var
#> [1] 6
#> 
#> $gamma
#> [1] 1
#> 
#> $beta
#> [1] 0
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Batch normalization (per-feature standardisation then affine rescale)"
#>