Momentum step (Geron Ch 11, morie.fn hmmom)

v <- beta v + g; theta <- theta - eta v. On a constant gradient the step approaches eta g / (1 - beta), which is the terminal_step field: momentum effectively multiplies the learning rate.

Usage

morie_geron_momentum(
  grads,
  v = NULL,
  beta = 0.9,
  eta = 0.01,
  theta = NULL,
  nesterov = FALSE
)

Arguments

grads

Gradient vector.

v

Velocity; default zeros.

beta

Decay in [0, 1).

eta

Positive rate.

theta

Parameters; default zeros.

nesterov

Look-ahead form.

Value

List with theta, theta_next, step, v, terminal_step, beta, estimate, n.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_momentum(V)
#> $theta
#> [1] -0.01 -0.02 -0.03 -0.04 -0.05 -0.06 -0.07 -0.08
#> 
#> $theta_next
#> [1] -0.01 -0.02 -0.03 -0.04 -0.05 -0.06 -0.07 -0.08
#> 
#> $step
#> [1] -0.01 -0.02 -0.03 -0.04 -0.05 -0.06 -0.07 -0.08
#> 
#> $v
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $terminal_step
#> [1] 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
#> 
#> $beta
#> [1] 0.9
#> 
#> $estimate
#> [1] 0.1428286
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Momentum optimization step"
#>