Backpropagation through time (Geron Ch 13, morie.fn grbptt)

delta_t = (dL/dh_t + W_h delta_{t+1}) * (1 - h_t^2); the shared weight picks up a term at every step, which is where the vanishing gradient shows up in per_step_delta_norm.

Usage

morie_geron_backprop_through_time(
  loss_grads,
  hiddens,
  inputs,
  W_h = NULL,
  h_init = NULL
)

Arguments

loss_grads, hiddens

Matrices (T, H).

inputs

Matrix (T, D).

W_h

Optional recurrent weights (H, H).

h_init

Optional state before step 1.

Value

List with grad_Wx, grad_Wh, grad_b, deltas, per_step_delta_norm, vanishing_ratio.

Examples

morie_geron_backprop_through_time(matrix(c(1, 0.5, 0.5, 1), 2,
    2, byrow = TRUE), matrix(c(0.2, -0.1, 0.4, 0.3), 2, 2, byrow = TRUE),
    matrix(c(1, 0, 0, 2), 2, 2, byrow = TRUE), W_h = matrix(c(0.5,
        0.1, -0.2, 0.3), 2, 2, byrow = TRUE), h_init = c(0.1,
        0.2))
#> $grad_Wx
#>         [,1]    [,2]
#> [1,] 1.24896 0.68211
#> [2,] 0.84000 1.82000
#> 
#> $grad_Wh
#>          [,1]     [,2]
#> [1,] 0.208896 0.250211
#> [2,] 0.207792 0.045422
#> 
#> $grad_b
#> [1] 1.66896 1.59211
#> 
#> $deltas
#>         [,1]    [,2]
#> [1,] 1.24896 0.68211
#> [2,] 0.42000 0.91000
#> 
#> $per_step_delta_norm
#> [1] 1.423086 1.002247
#> 
#> $vanishing_ratio
#> [1] 1.419895
#> 
#> $estimate
#> [1] 0.3892091
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Backpropagation through time (tanh RNN)"
#>