Nesterov accelerated gradient step — morie

The look-ahead form of Sutskever et al. (2013): the velocity updates as for heavy-ball but the parameter step is mu * v - lr * g. For the exact method g should be evaluated at the look-ahead point theta + mu * v.

Usage

morie_nesterov(g, mu = 0.9, lr = 0.01, state = NULL)

Arguments

g: Gradient (see note on where to evaluate it).
mu: Momentum coefficient in [0, 1).
lr: Step size.
state: List from the previous call, or NULL.

Value

List with update, state, v, t, step_norm.

References

Nesterov, Y. (1983). Soviet Mathematics Doklady 27(2), 372-376. Sutskever, I., et al. (2013). ICML 2013, 1139-1147.

Examples

round(morie_nesterov(1, mu = 0.9, lr = 0.1)$update, 6)
#> [1] -0.19