Layer normalization inside an RNN cell (Geron Ch 13, morie.fn hmlnr)

Layer normalization inside an RNN cell (Geron Ch 13, morie.fn hmlnr)

Usage

morie_geron_layer_norm_rnn(
  x,
  gamma = 1,
  beta = 0,
  eps = 1e-05,
  activation = "tanh"
)

Arguments

x

Pre-activations (T, n_units).

gamma, beta

Scale/shift.

eps

Variance floor.

activation

tanh/relu/none.

Value

List with h, normalized, mu, var, estimate, n, method.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_layer_norm_rnn(V)
#> $h
#>            [,1]       [,2]      [,3]       [,4]      [,5]     [,6]      [,7]
#> [1,] -0.9099999 -0.7972751 -0.574794 -0.2148186 0.2148186 0.574794 0.7972751
#>           [,8]
#> [1,] 0.9099999
#> 
#> $normalized
#>           [,1]      [,2]      [,3]       [,4]      [,5]     [,6]     [,7]
#> [1,] -1.527524 -1.091088 -0.654653 -0.2182177 0.2182177 0.654653 1.091088
#>          [,8]
#> [1,] 1.527524
#> 
#> $pre_activation
#>           [,1]      [,2]      [,3]       [,4]      [,5]     [,6]     [,7]
#> [1,] -1.527524 -1.091088 -0.654653 -0.2182177 0.2182177 0.654653 1.091088
#>          [,8]
#> [1,] 1.527524
#> 
#> $mu
#> [1] 4.5
#> 
#> $var
#> [1] 5.25
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Layer normalization inside an RNN cell"
#>