Rescales each layer's step by the trust ratio
eta * ||w|| / (||g|| + wd * ||w||), so the step length tracks the weight
norm. A zero weight norm falls back to an unscaled step.
Usage
morie_lars(
g,
w,
lr = 0.1,
mu = 0.9,
wd = 0,
eta = 0.001,
eps = 1e-08,
state = NULL
)
Arguments
- g
Gradient for this layer.
- w
Current weights for this layer, same length as g.
- lr
Global learning rate.
- mu
Momentum coefficient in [0, 1).
- wd
Weight decay, folded into the gradient (not decoupled).
- eta
Trust coefficient.
- eps
Denominator floor.
- state
List from the previous call, or NULL.
Value
List with update, state, trust_ratio, w_norm, g_norm, v.
References
You, Y., Gitman, I., & Ginsburg, B. (2017). arXiv:1708.03888.
Examples
morie_lars(c(1, 1), c(0, 0), lr = 0.1, mu = 0)$trust_ratio
#> [1] 1
