Bootstrap .632 / .632+ validation

Usage

bootstrap_validate(
  fit_fn,
  predict_fn,
  X,
  y,
  n_bootstraps = 200L,
  scoring = "roc_auc",
  method = "632plus",
  random_state = 42L
)

Arguments

fit_fn: A function (X, y) -> model.
predict_fn: A function (model, X) -> probability vector.
X: Matrix or data frame of features.
y: Vector of targets.
n_bootstraps: Number of bootstrap replicates.
scoring: "roc_auc", "accuracy", "brier".
method: "632" or "632plus".
random_state: Seed.

Value

An object of class "class_name".

Examples

set.seed(1)
X <- matrix(rnorm(120 * 3), 120, 3)
y <- as.integer(plogis(X[, 1] - 0.5 * X[, 2]) > runif(120))
fit <- function(X, y) suppressWarnings(
  glm(y ~ ., data = data.frame(y = y, X), family = binomial()))
pred <- function(m, X) predict(m, newdata = data.frame(X), type = "response")
res <- bootstrap_validate(fit, pred, X, y, n_bootstraps = 20L,
                          method = "632", scoring = "accuracy")
res$mean
#> [1] 0.7421005