Single-hidden-layer neural network

The ESL Sec 11.3 architecture, fitted by full-batch gradient descent. Weights are initialised small but NON-ZERO: exactly zero leaves the model perfectly symmetric, so every hidden unit computes the same thing and stays that way, while large starting weights saturate the sigmoid and kill the gradient. Inputs are standardised by default, because weight decay is otherwise a statement about the predictors' measurement units. The penalty spares the biases, since shrinking an intercept toward zero is a claim about the origin.

Usage

morie_esl_neural_net(
  X,
  y,
  M = 5L,
  lambda_ = 0,
  lr = 0.1,
  n_epochs = 400L,
  task = "regression",
  newdata = NULL,
  seed = 0L,
  standardize = TRUE
)

Arguments

X

Predictors, n by p.

y

Response; numeric for regression, class labels otherwise.

M

Hidden units.

lambda_

Weight decay, non-negative.

lr

Learning rate.

n_epochs

Full-batch gradient steps.

task

"regression" or "classification".

newdata

Points to predict at; defaults to X.

seed

Seed for weight initialisation.

standardize

Standardise the inputs.

Value

List with fitted (or prob/class), alpha, beta, loss_path, hidden, and r_squared or accuracy.

References

Hastie, T., et al. (2009). ESL (2nd ed.), Sec 11.3-11.4. Springer.

Examples

set.seed(1)
X <- matrix(runif(400, -2, 2), ncol = 2)
y <- sin(X[, 1]) + X[, 2]^2
morie_esl_neural_net(X, y,
  M = 8, lr = 0.3, n_epochs = 2000L,
  seed = 1L
)$r_squared > 0.8
#> [1] TRUE