AdamW with decoupled weight decay (Geron Ch 11, morie.fn grwdc)

theta -= eta (m_hat/(sqrt(s_hat)+eps) + lam theta); the decay sits OUTSIDE the adaptive normalisation. The Adam part comes from morie_geron_adam_update().

Usage

morie_geron_adamw_decoupled_weight_decay(
  theta,
  grad,
  m,
  s,
  t,
  eta,
  b1 = 0.9,
  b2 = 0.999,
  eps = 1e-08,
  lam = 0.01
)

Arguments

theta, grad, m, s

Parameters, gradient and moment accumulators.

t

1-based timestep.

eta

Learning rate.

b1, b2, eps

Adam hyperparameters.

lam

Non-negative decoupled decay.

Value

List with theta_new, m_new, s_new, adam_step, decay_step, step.

Examples

morie_geron_adamw_decoupled_weight_decay(c(1, -2), c(0.5, 0.25),
    c(0, 0), c(0, 0), 3, 0.01, lam = 0.05)
#> $theta_new
#> [1]  0.9931119 -2.0053881
#> 
#> $m_new
#> [1] 0.050 0.025
#> 
#> $s_new
#> [1] 2.50e-04 6.25e-05
#> 
#> $adam_step
#> [1] 0.006388136 0.006388136
#> 
#> $decay_step
#> [1]  5e-04 -1e-03
#> 
#> $step
#> [1] 0.006888136 0.005388136
#> 
#> $t
#> [1] 3
#> 
#> $estimate
#> [1] 0.008745194
#> 
#> $n
#> [1] 2
#>