Repeated K-fold cross-validation

Repeats .boot_cross_validate() n_repeats times with different RNG seeds and pools the per-fold scores. caret::trainControl(method = "repeatedcv") and rsample::vfold_cv both implement the same partitioning (cross-referenced).

Usage

repeated_cv(
  X,
  y,
  model_fn,
  score_fn,
  n_folds = 10L,
  n_repeats = 10L,
  seed = 42L
)

Arguments

X

Numeric matrix or data.frame of predictors.

y

Numeric or factor outcome vector aligned with rows of X.

model_fn

Function (X, y) -> fitted-model used on each training fold.

score_fn

Function (y_true, y_pred) -> numeric returning a single performance metric.

n_folds

Integer; number of folds per repeat (default 10).

n_repeats

Number of repetitions.

seed

Integer RNG seed for reproducibility.

Value

A morie_cv_result pooling scores across repeats.

See also

caret::trainControl, rsample::vfold_cv.

Examples

if (requireNamespace("caret", quietly = TRUE) && requireNamespace("ranger", quietly = TRUE) &&
  requireNamespace("rsample", quietly = TRUE)) {
  set.seed(1)
  X <- matrix(rnorm(80), ncol = 2)
  y <- rnorm(40)
  predict.lm_lite3 <- function(object, newdata, ...) {
    drop(cbind(1, newdata) %*% object$coef)
  }
  registerS3method("predict", "lm_lite3", predict.lm_lite3)
  res <- repeated_cv(X, y,
    model_fn = function(Xt, yt) {
      fit <- stats::lm.fit(cbind(1, Xt), yt)
      structure(list(coef = fit$coefficients), class = "lm_lite3")
    },
    score_fn = function(yt, yp) mean((yt - yp)^2),
    n_folds = 5L, n_repeats = 2L
  )
  str(res, max.level = 1)
}
#> List of 8
#>  $ scores    : num [1:10] 0.498 0.963 0.549 0.586 0.797 ...
#>  $ mean_score: num 0.684
#>  $ se_score  : num 0.0631
#>  $ ci_lower  : num 0.561
#>  $ ci_upper  : num 0.808
#>  $ n_folds   : int 10
#>  $ metric    : chr "custom"
#>  $ fold_sizes: int [1:10] 8 8 8 8 8 8 8 8 8 8
#>  - attr(*, "class")= chr [1:2] "morie_cv_result" "list"