Bootstrap optimism-corrected performance

Usage

detect_overfitting(
  fit_fn,
  predict_fn,
  X,
  y,
  scoring = "roc_auc",
  n_bootstrap = 200L,
  random_state = 42L
)

Arguments

fit_fn: A function (X, y) -> model.
predict_fn: A function (model, X) -> probability vector.
X: Matrix or data frame of features.
y: Vector of targets.
scoring: "roc_auc", "accuracy", "brier".
n_bootstrap: Integer; number of bootstrap resamples used to estimate the optimism correction (default 200).
random_state: Seed.

Value

An object of class "class_name".

Examples

set.seed(1)
X <- matrix(rnorm(120 * 3), 120, 3)
y <- as.integer(plogis(X[, 1] - 0.5 * X[, 2]) > runif(120))
fit <- function(X, y) suppressWarnings(
  glm(y ~ ., data = data.frame(y = y, X), family = binomial()))
pred <- function(m, X) predict(m, newdata = data.frame(X), type = "response")
res <- detect_overfitting(fit, pred, X, y, n_bootstrap = 20L,
                          scoring = "accuracy")
res$recommendation
#> [1] "Moderate optimism. Bootstrap-corrected estimates recommended."