LayerNorm per-token normalization

Normalises each instance across its features, then applies a learned gain and bias: y equals gamma times (x - mu) / sqrt(sigma2 + eps) plus beta, with eps inside the square root. Statistics come from a single training case, which is what distinguishes layer norm from batch norm. There is exactly one implementation: this function delegates to morie_geron_layer_normalization.

Usage

Layrnm(X, gamma = 1, beta = 0, eps = 1e-05)

Arguments

X

Numeric vector or matrix, shape d or m by d.

gamma

Learned gain, scalar or length d. Default 1.

beta

Learned shift, scalar or length d. Default 0.

eps

Positive stabiliser inside the square root. Default 1e-5.

Value

List with output, normalized, mean, variance, estimate, n, method.

References

Ba, J. L., Kiros, J. R. and Hinton, G. E. (2016), Layer Normalization, arXiv:1607.06450, Section 3.

Examples

Layrnm(c(1, 3), eps = 0)$normalized
#> [1] -1  1