Grid search with cross-validation (R parity)

Native k-fold cross-validated grid search. Every candidate in tune_grid is fitted on each training fold with one of rmorie's own learners and scored on the held-out fold; the best candidate is the one with the lowest mean RMSE (regression) or highest mean accuracy (classification). The resampling protocol reproduces caret::train(..., trControl = trainControl(method = "cv")): after set.seed(seed) one integer is drawn to seed the fold assignment, which is made with caret's stratified createFolds algorithm (outcome classes, or up to five quantile bins of a numeric outcome, are spread evenly over the folds) in an isolated RNG stream; per-fold RMSE / R-squared / MAE or accuracy / Cohen's kappa are averaged over the folds (NA folds dropped), and ties are broken as caret does. With the same seed the folds, and therefore the scores, agree with caret's.

Usage

morie_grid_search_cv(
  x,
  y,
  method = NULL,
  tune_grid = NULL,
  cv = 5L,
  task = "auto",
  seed = 0L
)

Arguments

x

Numeric predictor matrix.

y

Response.

method

Learner id, one of "lm", "glm", "glmnet", "ridge" (default "glmnet" for classification, "ridge" for regression).

tune_grid

data.frame of hyperparameter combos to evaluate (column names are the method's tuning parameters). When NULL and method is supplied, a three-level default grid is built as caret does.

cv

CV folds.

task

"auto", "classification", or "regression".

seed

RNG seed (drives the fold assignment; the caller's RNG stream is restored on exit).

Value

Named list: estimate (CV score of the best candidate), best_params, best_score (same as estimate), cv_results_params (the candidate parameters, sorted, together with the fold standard deviations of the metrics), cv_results_mean_score (mean CV RMSE for regression, accuracy for classification, aligned with cv_results_params), task, n, method.

Details

Supported method values and their native learners:

"lm"

least squares (pivoted QR), tuning parameter intercept (logical); regression only.

"glm"

logistic (classification, two classes) or Gaussian (regression) GLM by IRLS (stats::glm.fit); tuning parameter parameter = "none".

"glmnet"

elastic net by pathwise coordinate descent on standardised predictors (Friedman, Hastie and Tibshirani 2010): Gaussian, binomial or multinomial by task. As in glmnet, each training fold is fitted along glmnet's default 100-point lambda path (same lambda_max, lambda.min.ratio and early-stopping rules) and predictions at a grid lambda are linearly interpolated between path points. Tuning parameters alpha, lambda.

"ridge"

ridge regression in the Zou-Hastie elastic net parametrisation of elasticnet::enet at fraction 1 (unit-norm predictors, coefficients rescaled by 1 + lambda); tuning parameter lambda; regression only.

Any other method is an error naming the supported set.

References

Friedman, J., Hastie, T. and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software 33(1). Kuhn, M. (2008). Building predictive models in R using the caret package. Journal of Statistical Software 28(5).

Examples

set.seed(1)
morie_grid_search_cv(
  x = matrix(rnorm(150), 50, 3), y = rnorm(50),
  method = "lm", tune_grid = data.frame(intercept = c(TRUE, FALSE)),
  cv = 3L, task = "regression", seed = 1L
)
#> $estimate
#> [1] 1.785965e-16
#> 
#> $best_params
#> $best_params$intercept
#> [1] TRUE
#> 
#> 
#> $best_score
#> [1] 1.785965e-16
#> 
#> $cv_results_params
#>   intercept       RMSESD  RsquaredSD        MAESD
#> 1     FALSE 2.072260e-16 1.35974e-16 1.660726e-16
#> 2      TRUE 2.055364e-16 1.35974e-16 1.622891e-16
#> 
#> $cv_results_mean_score
#> [1] 1.809802e-16 1.785965e-16
#> 
#> $task
#> [1] "regression"
#> 
#> $n
#> [1] 50
#> 
#> $method
#> [1] "Grid search CV (lm)"
#>