Repeated K-fold cross-validation

Repeats .boot_cross_validate() n_repeats times with different RNG seeds and pools the per-fold scores. caret::trainControl(method = "repeatedcv") and rsample::vfold_cv both implement the same partitioning (cross-referenced).

Usage

repeated_cv(
  X,
  y,
  model_fn,
  score_fn,
  n_folds = 10L,
  n_repeats = 10L,
  seed = 42L
)

Arguments

X

Numeric matrix or data.frame of predictors.

y

Numeric or factor outcome vector aligned with rows of X.

model_fn

Function (X, y) -> fitted-model used on each training fold.

score_fn

Function (y_true, y_pred) -> numeric returning a single performance metric.

n_folds

Integer; number of folds per repeat (default 10).

n_repeats

Number of repetitions.

seed

Integer RNG seed for reproducibility.

Value

A morie_cv_result pooling scores across repeats.

See also

caret::trainControl, rsample::vfold_cv.

Examples

set.seed(1)
X <- matrix(rnorm(80), ncol = 2); y <- rnorm(40)
predict.lm_lite3 <- function(object, newdata, ...) {
  drop(cbind(1, newdata) %*% object$coef)
}
registerS3method("predict", "lm_lite3", predict.lm_lite3)
res <- repeated_cv(X, y,
  model_fn = function(Xt, yt) {
    fit <- stats::lm.fit(cbind(1, Xt), yt)
    structure(list(coef = fit$coefficients), class = "lm_lite3")
  },
  score_fn = function(yt, yp) mean((yt - yp)^2),
  n_folds = 5L, n_repeats = 2L)
str(res, max.level = 1)
#> List of 8
#>  $ scores    : num [1:10] 0.332 0.865 0.705 0.931 1.147 ...
#>  $ mean_score: num 0.755
#>  $ se_score  : num 0.0707
#>  $ ci_lower  : num 0.617
#>  $ ci_upper  : num 0.894
#>  $ n_folds   : int 10
#>  $ metric    : chr "custom"
#>  $ fold_sizes: int [1:10] 8 8 8 8 8 8 8 8 8 8
#>  - attr(*, "class")= chr [1:2] "morie_cv_result" "list"