Stochastic gradient descent (Geron Ch 4, morie.fn grsgd)

One LCG-drawn sample per step, gradient 2 x_i (x_i^T theta - y_i); passing both t0 and t1 anneals the rate as t0/(t + t1). The sampled row indices are 0-based.

Usage

morie_geron_stochastic_gradient_descent(
  X,
  y,
  theta,
  eta,
  n_iter,
  seed = 42,
  t0 = NULL,
  t1 = NULL
)

Arguments

X, y, theta

Data and starting parameters.

eta

Fixed rate (ignored when scheduled).

n_iter

Number of steps.

seed

LCG seed.

t0, t1

Optional schedule constants, supplied together.

Value

List with theta, path, cost_path, learning_rates, sample_order.

Examples

X <- matrix(c(1, 2, 1, 3, 1, 5, 1, 7), 4, byrow = TRUE)
y <- c(2, 3, 6, 9)
morie_geron_stochastic_gradient_descent(X, y, c(0, 0), 0.01,
    6, seed = 23)
#> $theta
#> [1] 0.2833821 1.2452311
#> 
#> $path
#> $path[[1]]
#> [1] 0.04 0.08
#> 
#> $path[[2]]
#> [1] 0.0944 0.2432
#> 
#> $path[[3]]
#> [1] 0.188192 0.712160
#> 
#> $path[[4]]
#> [1] 0.2332122 0.9372608
#> 
#> $path[[5]]
#> [1] 0.2548218 1.0453092
#> 
#> $path[[6]]
#> [1] 0.2833821 1.2452311
#> 
#> 
#> $cost_path
#> [1] 32.5000000 28.0280000 20.1568858  5.0791982  1.5118365  0.6450543  0.4742447
#> 
#> $learning_rates
#> [1] 0.01 0.01 0.01 0.01 0.01 0.01
#> 
#> $sample_order
#> [1] 0 1 2 2 2 3
#> 
#> $estimate
#> [1] 0.2833821 1.2452311
#> 
#> $n
#> [1] 4
#>