Simple (Elman) RNN cell (Geron Ch 15, morie.fn grrnnc)

h_t = tanh(Whh h_{t-1} + Wxh x_t + b); the spectral norm of Whh decides vanishing versus exploding gradients.

Usage

morie_geron_simple_rnn_cell(x_t, h_prev, Whh, Wxh, b)

Arguments

x_t, h_prev

State vectors.

Whh

(H, H) recurrent weights.

Wxh

(H, n) input weights.

b

Length-H bias.

Value

List with h, pre_activation, derivative, spectral_norm_Whh, saturated.

Examples

set.seed(4)
morie_geron_simple_rnn_cell(x_t = c(0.5, -0.2), h_prev = c(0.1, 0.3, 0),
                            Whh = matrix(0.2, 3, 3),
                            Wxh = matrix(0.1, 3, 2), b = rep(0, 3))
#> $h
#> [1] 0.1095585 0.1095585 0.1095585
#> 
#> $pre_activation
#> [1] 0.11 0.11 0.11
#> 
#> $derivative
#> [1] 0.9879969 0.9879969 0.9879969
#> 
#> $spectral_norm_Whh
#> [1] 0.6
#> 
#> $saturated
#> [1] 0
#> 
#> $estimate
#> [1] 0.1095585 0.1095585 0.1095585
#> 
#> $n
#> [1] 3
#>