NAdam step (Geron Ch 11, morie.fn hmnadm)

Adam moments with the Nesterov-corrected first moment m_hat = b1 m / (1 - b1^(t+1)) + (1 - b1) g / (1 - b1^t). Note the t+1 exponent in the first term: it is not a typo, it is what pulls the current gradient into the step without a second evaluation.

Usage

morie_geron_nadam(
  grads,
  m = NULL,
  v = NULL,
  b1 = 0.9,
  b2 = 0.999,
  eta = 0.001,
  t = 1,
  eps = 1e-08,
  theta = NULL
)

Arguments

grads

Gradient vector.

m, v

Moments; default zeros.

b1, b2

Decays in [0, 1).

eta

Positive rate.

t

1-based timestep.

eps

Non-negative floor.

theta

Parameters; default zeros.

Value

List with theta, theta_next, step, m, v, m_hat, v_hat, t, estimate, n.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_nadam(V)
#> $theta
#> [1] -0.001473684 -0.001473684 -0.001473684 -0.001473684 -0.001473684
#> [6] -0.001473684 -0.001473684 -0.001473684
#> 
#> $theta_next
#> [1] -0.001473684 -0.001473684 -0.001473684 -0.001473684 -0.001473684
#> [6] -0.001473684 -0.001473684 -0.001473684
#> 
#> $step
#> [1] -0.001473684 -0.001473684 -0.001473684 -0.001473684 -0.001473684
#> [6] -0.001473684 -0.001473684 -0.001473684
#> 
#> $m
#> [1] 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
#> 
#> $v
#> [1] 0.001 0.004 0.009 0.016 0.025 0.036 0.049 0.064
#> 
#> $m_hat
#> [1]  1.473684  2.947368  4.421053  5.894737  7.368421  8.842105 10.315789
#> [8] 11.789474
#> 
#> $v_hat
#> [1]  1  4  9 16 25 36 49 64
#> 
#> $t
#> [1] 1
#> 
#> $estimate
#> [1] 0.004168208
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "NAdam (Adam moments with a Nesterov-corrected first moment)"
#>