AdaMax step (Geron Ch 11, morie.fn hmadmx)

u <- max(b2 u, |g|); only m is bias-corrected, because an exponentially weighted max needs no correction.

Usage

morie_geron_adamax(
  grads,
  m = NULL,
  u = NULL,
  b1 = 0.9,
  b2 = 0.999,
  eta = 0.002,
  t = 1,
  theta = NULL
)

Arguments

grads

Gradient vector.

m

First moment.

u

Infinity-norm.

b1, b2

Decay rates.

eta

Learning rate.

t

Timestep.

theta

Parameters.

Value

List with theta, step, m, u, m_hat.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_adamax(V)
#> $theta
#> [1] -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002
#> 
#> $theta_next
#> [1] -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002
#> 
#> $step
#> [1] -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002 -0.002
#> 
#> $m
#> [1] 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
#> 
#> $u
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $m_hat
#> [1] 1 2 3 4 5 6 7 8
#> 
#> $t
#> [1] 1
#> 
#> $estimate
#> [1] 0.005656854
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "AdaMax (Adam with an L-infinity second moment)"
#>