Cross-validate a model using a user-supplied fit/predict pair

Usage

cross_validate(
  fit_fn,
  predict_fn,
  X,
  y,
  method = "stratified_kfold",
  n_folds = 5L,
  n_repeats = 10L,
  scoring = "roc_auc",
  groups = NULL,
  confidence = 0.95,
  random_state = 42L
)

Arguments

fit_fn: A function (X, y) -> model.
predict_fn: A function (model, X) -> probability vector.
X: Matrix or data frame of features.
y: Vector of targets.
method: Resampling strategy: "kfold", "stratified_kfold", "grouped_kfold", "loo", "monte_carlo", "time_series".
n_folds: Number of folds.
n_repeats: Repeats for monte_carlo.
scoring: "roc_auc", "accuracy", "brier".
groups: Group labels for grouped_kfold.
confidence: Confidence level for the score CI.
random_state: Seed.

Value

An object of class "class_name".

Examples

set.seed(1)
X <- matrix(rnorm(200 * 3), 200, 3)
y <- as.integer(plogis(X[, 1] - 0.5 * X[, 2]) > runif(200))
fit <- function(X, y) suppressWarnings(
  glm(y ~ ., data = data.frame(y = y, X), family = binomial()))
pred <- function(m, X) predict(m, newdata = data.frame(X), type = "response")
res <- cross_validate(fit, pred, X, y, method = "kfold", n_folds = 5L)
res$mean
#> [1] 0.7715926