The .632 and .632+ prediction-error estimators

ESL Eq. (7.57), Err^(.632) = .368 err_bar + .632 Err^(1), and Eq. (7.61), Err^(.632+) = (1-w) err_bar + w Err^(1) with w = .632 / (1 - .368 R).

Usage

morie_esl_oob_632(
  err_train,
  err_loo_boot,
  gamma = NULL,
  y = NULL,
  y_pred = NULL,
  p1 = NULL,
  q1 = NULL
)

Arguments

err_train

the training error err_bar.

err_loo_boot

the leave-one-out bootstrap error (7.56).

gamma

optional no-information error rate.

y, y_pred

optional response and fitted values, for (7.58).

p1, q1

optional proportions, for (7.59).

Value

list: value, err_632, err_632_plus, weight, relative_overfitting_rate, gamma, err_train, err_loo_boot, uses_leave_one_out, method.

Details

The second argument is the LEAVE-ONE-OUT bootstrap (7.56), not the naive (7.54). That distinction is the point of the section: (7.54) is biased downward by train/test overlap, while (7.56) is biased upward because a bootstrap sample holds only about 0.632N distinct points. The .632 constant is the inclusion probability of (7.55). Passing (7.54) here applies a downward correction to a quantity that was already too small.

.632 breaks down on overfit rules. The book's case: a 1-nearest-neighbour rule on independent labels has err_bar = 0 and Err^(1) = 0.5, giving .632 * 0.5 = 0.316 against a true rate of 0.5. .632+ repairs it using the no-information error rate gamma – (7.58) gamma = N^-2 sum_i sum_i' L(y_i, f(x_i')) or (7.59) p1(1-q1) + (1-p1)q1 – through the relative overfitting rate (7.60) R = (Err^(1) - err_bar)/(gamma - err_bar). On that example R = w = 1 and it returns 0.5.

References

Hastie, Tibshirani and Friedman (2009), Sec. 7.11, Eqs. (7.55)-(7.61); Efron (1983); Efron and Tibshirani (1997).

Examples

# the book's 1-NN counterexample
morie_esl_oob_632(0, 0.5, gamma = 0.5)$err_632_plus
#> [1] 0.5