The LARS trust ratio applied to Adam's direction rather than to the raw
gradient. A zero weight or direction norm falls back to a trust ratio of 1.
Usage
morie_lamb(
g,
w,
lr = 0.001,
beta1 = 0.9,
beta2 = 0.999,
wd = 0.01,
eps = 1e-06,
state = NULL
)
Arguments
- g
Gradient for this layer.
- w
Current weights for this layer, same length as g.
- lr
Global learning rate.
- beta1, beta2
Moment decay rates in [0, 1).
- wd
Weight decay applied to the Adam direction.
- eps
Denominator floor.
- state
List from the previous call, or NULL.
Value
List with update, state, trust_ratio, m, v, direction.
References
You, Y., Li, J., Reddi, S., et al. (2020). ICLR 2020.
Examples
morie_lamb(c(1, 1), c(0, 0), lr = 0.1, wd = 0)$trust_ratio
#> [1] 1
