Layer normalization (Geron Ch 15, morie.fn grln)

Per-row (x - mu)/sqrt(var + eps) with the POPULATION variance, then gamma / beta. No batch axis, unlike batch norm.

Usage

morie_geron_layer_normalization(X, gamma = 1, beta = 0, eps = 1e-05)

Arguments

X

Vector (one row) or (m, d) matrix, d >= 2.

gamma, beta

Scalars or length-d vectors.

eps

Non-negative variance floor.

Value

List with output, normalized, mean, variance.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_layer_normalization(V)
#> $output
#> [1] -1.5275238 -1.0910884 -0.6546530 -0.2182177  0.2182177  0.6546530  1.0910884
#> [8]  1.5275238
#> 
#> $normalized
#> [1] -1.5275238 -1.0910884 -0.6546530 -0.2182177  0.2182177  0.6546530  1.0910884
#> [8]  1.5275238
#> 
#> $mean
#> [1] 4.5
#> 
#> $variance
#> [1] 5.25
#> 
#> $eps
#> [1] 1e-05
#> 
#> $estimate
#> [1] -1.5275238 -1.0910884 -0.6546530 -0.2182177  0.2182177  0.6546530  1.0910884
#> [8]  1.5275238
#> 
#> $n
#> [1] 1
#>