Nesterov accelerated gradient (Geron Ch 11, morie.fn grnag)

Gradient evaluated at the look-ahead theta - eta beta v, then v = beta v + g and theta -= eta v.

Usage

morie_geron_nesterov_accelerated_gradient(
  theta,
  grad_fn,
  v,
  eta,
  beta = 0.9,
  n_steps = 1
)

Arguments

theta

Starting parameters.

grad_fn

function(theta) -> gradient.

v

Velocity, same length as theta.

eta

Positive learning rate.

beta

Momentum in [0, 1).

n_steps

Number of steps.

Value

List with theta_new, v_new, lookahead, gradient, path, step.

Examples

morie_geron_nesterov_accelerated_gradient(c(1, -1), function(t) 2 *
    t, c(0, 0), 0.1, 0.9, n_steps = 3)
#> $theta_new
#> [1]  0.17792 -0.17792
#> 
#> $v_new
#> [1]  3.1808 -3.1808
#> 
#> $lookahead
#> [1]  0.2224 -0.2224
#> 
#> $gradient
#> [1]  0.4448 -0.4448
#> 
#> $path
#> $path[[1]]
#> [1]  0.8 -0.8
#> 
#> $path[[2]]
#> [1]  0.496 -0.496
#> 
#> $path[[3]]
#> [1]  0.17792 -0.17792
#> 
#> 
#> $step
#> [1]  0.31808 -0.31808
#> 
#> $estimate
#> [1]  0.17792 -0.17792
#> 
#> $n
#> [1] 2
#>