Bias-corrected first and second moment estimates (Kingma & Ba, 2015).
The bias correction is what makes the first step approximately lr in
size whatever the gradient's scale.
Usage
morie_adam(
g,
beta1 = 0.9,
beta2 = 0.999,
lr = 0.001,
eps = 1e-08,
state = NULL
)
Arguments
- g
Gradient at the current parameters.
- beta1, beta2
Moment decay rates in [0, 1).
- lr
Step size.
- eps
Denominator floor.
- state
List returned by the previous call, or NULL to start fresh.
Value
List with update, state, m, v, t, step_norm.
References
Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic
optimization. ICLR 2015.
Examples
x <- 0; st <- NULL
for (i in 1:4000) {
r <- morie_adam(2 * (x - 3), lr = 0.05, state = st)
x <- x + r$update; st <- r$state
}
abs(x - 3) < 1e-3
#> [1] TRUE
