.632 and .632+ bootstrap estimators for prediction error

Computes apparent error, mean OOB bootstrap error, the .632 estimator (Efron 1983), and the .632+ no-information-adjusted estimator (Efron and Tibshirani 1997). \(ipred::errorest(..., estimator = "632plus")\) implements the same family in ipred; it is cross-referenced for users who already work with ipred's predict.\<learner\> ecosystem. The inline implementation is retained because rmorie's API takes naked model_fn / score_fn callables and is consumed by downstream MRM code.

Usage

bootstrap_632(X, y, model_fn, score_fn, n_boot = 200L, seed = 42L)

Arguments

X

Numeric design matrix (n x p).

y

Numeric response (length n).

model_fn

Function model_fn(X_train, y_train) returning a model object that supports predict(model, X_test).

score_fn

Function score_fn(y_true, y_pred) -> scalar.

n_boot

Number of bootstrap replicates.

seed

Random seed.

Value

Named numeric list with apparent_error, bootstrap_error, error_632, error_632plus.

See also

ipred::errorest.

Examples

if (requireNamespace("caret", quietly = TRUE) && requireNamespace("ranger", quietly = TRUE) &&
  requireNamespace("rsample", quietly = TRUE)) {
  set.seed(1)
  X <- matrix(rnorm(25 * 2), ncol = 2)
  y <- rnorm(25)
  model_fn <- function(Xt, yt) {
    structure(
      list(coef = drop(solve(crossprod(Xt), crossprod(Xt, yt)))),
      class = "lm_lite"
    )
  }
  predict.lm_lite <- function(object, newdata, ...) drop(newdata %*% object$coef)
  registerS3method("predict", "lm_lite", predict.lm_lite)
  score_fn <- function(yt, yp) mean((yt - yp)^2)
  res <- bootstrap_632(X, y, model_fn, score_fn, n_boot = 10L)
  res$error_632
}
#> Registered S3 method overwritten by 'pROC':
#>   method   from            
#>   plot.roc spatstat.explore
#> [1] 1.278641