Adam step (Geron Ch 11, morie.fn hmadam) — morie_geron

m <- b1 m + (1-b1) g; v <- b2 v + (1-b2) g^2; bias-correct with the 1-based timestep t; step = -eta m_hat / (sqrt(v_hat) + eps).

Usage

morie_geron_adam(
  grads,
  m = NULL,
  v = NULL,
  b1 = 0.9,
  b2 = 0.999,
  eta = 0.001,
  eps = 1e-08,
  t = 1,
  theta = NULL
)

Arguments

grads: Numeric gradient vector.
m, v: Moment accumulators; default zeros.
b1, b2: Decay rates in [0, 1).
eta: Positive learning rate.
eps: Non-negative floor.
t: 1-based timestep.
theta: Current parameters; default zeros.

Value

List with theta, step, m, v, m_hat, v_hat.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_adam(V)
#> $theta
#> [1] -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001
#> 
#> $theta_next
#> [1] -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001
#> 
#> $step
#> [1] -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001 -0.001
#> 
#> $m
#> [1] 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
#> 
#> $v
#> [1] 0.001 0.004 0.009 0.016 0.025 0.036 0.049 0.064
#> 
#> $m_hat
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $v_hat
#> [1]  1  4  9 16 25 36 49 64
#> 
#> $t
#> [1] 1
#> 
#> $estimate
#> [1] 0.002828427
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Adam (bias-corrected momentum + RMSProp)"
#>