Warm restarts: cosine decay with periodic restarts (SGDR)

eta = eta_min + 0.5*(eta_max-eta_min)*(1+cos(pi*T_cur/T_i)); cycle lengths grow geometrically T_{i+1} = round(T_i * factor).

Usage

morie_geron_warm_restarts(t, T0 = 10, factor = 2, eta_max = 0.1, eta_min = 0)

Arguments

t

Integer step or vector of steps (>=0).

T0

First cycle length.

factor

Geometric growth (>=1).

eta_max, eta_min

Learning-rate bounds.

Value

list with eta, cycle, cycle_length, step_in_cycle, restarts, T0, factor, estimate, n, method.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_warm_restarts(V)
#> $eta
#> [1] 0.09755283 0.09045085 0.07938926 0.06545085 0.05000000 0.03454915 0.02061074
#> [8] 0.00954915
#> 
#> $cycle
#> [1] 0 0 0 0 0 0 0 0
#> 
#> $cycle_length
#> [1] 10 10 10 10 10 10 10 10
#> 
#> $step_in_cycle
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $restarts
#> [1] 0
#> 
#> $T0
#> [1] 10
#> 
#> $factor
#> [1] 2
#> 
#> $estimate
#> [1] 0.00954915
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "SGDR: cosine annealing within geometrically growing cycles"
#>