AdaGrad optimiser step — morie

Accumulates the undecayed sum of squared gradients, so the effective step size is monotonically non-increasing and eventually stalls – the behaviour RMSProp's decay was introduced to fix.

Usage

morie_adagrad(g, lr = 0.01, eps = 1e-08, state = NULL)

Arguments

g: Gradient at the current parameters.
lr: Base step size.
eps: Denominator floor.
state: List from the previous call, or NULL.

Value

List with update, state, G, t, step_norm.

References

Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. JMLR 12, 2121-2159.

Examples

r1 <- morie_adagrad(1, lr = 0.1)
r2 <- morie_adagrad(1, lr = 0.1, state = r1$state)
abs(r2$update) < abs(r1$update)
#> [1] TRUE