Hoeffding confidence bound for a cross-validated risk

The cross-validated risk averages n held-out losses; for losses between 0 and \(B\) Hoeffding's inequality gives the half-width \(t = B \sqrt{\log(2/\delta) / (2n)}\). \(B\) is the known range of the loss, not the spread of the fold risks. Identical to the Python arm morie.fn.cvbnd.cvbnd.

Usage

cvbnd(cv_risks, n, n_folds = 5, delta = 0.05, loss_bound = 1)

Arguments

cv_risks

Numeric vector of per-fold risks.

n

Total sample size.

n_folds

Number of folds (reported only).

delta

Confidence parameter.

loss_bound

Upper bound of the loss.

Value

A list with cv_risk, cv_se, upper_bound, lower_bound, bound_width, loss_bound, n and delta.

References

Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association 58, 13-30.

Examples

cvbnd(c(0.10, 0.14, 0.12, 0.08, 0.11), n = 500)$bound_width
#> [1] 0.06073615