AdaGrad step, negative-step convention (Geron Ch 11, morie.fn hmadgr)

s <- s + g^2; step = -eta g / (sqrt(s) + eps).

Usage

morie_geron_adagrad(grads, s = NULL, eta = 0.01, eps = 1e-10, theta = NULL)

Arguments

grads

Gradient vector.

s

Squared-gradient accumulator.

eta

Learning rate.

eps

Floor.

theta

Parameters.

Value

List with theta, step, s, effective_lr.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_adagrad(V)
#> $theta
#> [1] -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01
#> 
#> $theta_next
#> [1] -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01
#> 
#> $step
#> [1] -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01 -0.01
#> 
#> $s
#> [1]  1  4  9 16 25 36 49 64
#> 
#> $effective_lr
#> [1] 0.010000000 0.005000000 0.003333333 0.002500000 0.002000000 0.001666667
#> [7] 0.001428571 0.001250000
#> 
#> $estimate
#> [1] 0.02828427
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "AdaGrad (per-parameter rates from accumulated squared gradients)"
#>