Skip to contents

1cycle learning-rate and momentum schedule (Geron Ch 11, morie.fn gr1cy)

Source: R/geron_ml_native.R
morie_geron_1cycle_schedule.Rd

Peak step is (T - 1) %/% 2; the LR is a linear interpolation over knots (0, peak, T-1) with values (eta_min, eta_max, eta_min) and the momentum mirrors it. T = 2 collapses the peak onto step 0, which is the two-knot branch.

Usage

morie_geron_1cycle_schedule(
  eta_min,
  eta_max,
  t,
  T,
  mom_max = 0.95,
  mom_min = 0.85
)

Arguments

eta_min, eta_max

Learning-rate bounds, 0 < eta_min < eta_max.

t

Reporting step, 0-based, 0 <= t < T.

T

Total steps, at least 2.

mom_max, mom_min

Momentum bounds, 0 <= mom_min < mom_max < 1.

Value

List with lr_schedule, momentum_schedule, peak_step, lr_at_t, momentum_at_t.

Examples

morie_geron_1cycle_schedule(0.1, 0.5, t = 1, T = 7)
#> $lr_schedule
#> [1] 0.1000000 0.2333333 0.3666667 0.5000000 0.3666667 0.2333333 0.1000000
#> 
#> $momentum_schedule
#> [1] 0.9500000 0.9166667 0.8833333 0.8500000 0.8833333 0.9166667 0.9500000
#> 
#> $peak_step
#> [1] 3
#> 
#> $lr_at_t
#> [1] 0.2333333
#> 
#> $momentum_at_t
#> [1] 0.9166667
#> 
#> $eta_min
#> [1] 0.1
#> 
#> $eta_max
#> [1] 0.5
#> 
#> $estimate
#> [1] 0.2333333
#> 
#> $n
#> [1] 7
#> 
#> $method
#> [1] "1cycle learning-rate schedule (Smith 2018)"
#>