Layer normalisation

Formula: mu = (1/H) sum_i a_i, sigma = sqrt((1/H) sum_i (a_i - mu)^2), y = gamma (a - mu)/sigma + beta

Usage

Vitlnorm(x, gamma = NULL, beta = NULL)

Arguments

x

See Usage.

gamma

See Usage.

beta

See Usage.

Value

List with y, mu, sigma, n, H.

References

Ba, Kiros and Hinton (2016), Layer Normalization, arXiv:1607.06450, equation (3). Verified against the paper.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
Vitlnorm(V)
#> $y
#>           [,1]      [,2]       [,3]       [,4]      [,5]      [,6]     [,7]
#> [1,] -1.527525 -1.091089 -0.6546537 -0.2182179 0.2182179 0.6546537 1.091089
#>          [,8]
#> [1,] 1.527525
#> 
#> $mu
#> [1] 4.5
#> 
#> $sigma
#> [1] 2.291288
#> 
#> $n
#> [1] 1
#> 
#> $H
#> [1] 8
#> 
#> $method
#> [1] "Layer normalisation"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"