Random forest for regression

ESL Algorithm 15.1. For b = 1..B: draw a bootstrap sample of size N, and grow a tree by repeating, at each terminal node, select mtry of the p variables at random, pick the best variable and split-point among those, and split. Predict by f_rf(x) = (1/B) sum_b T_b(x).

Usage

morie_esl_random_forest(
  X,
  y,
  B = 100,
  mtry = NULL,
  max_depth = 8L,
  min_node = 5L,
  newdata = NULL,
  seed = 0
)

Arguments

X

predictors.

y

numeric response.

B

number of trees.

mtry

variables sampled per node; floor(p/3) when NULL.

max_depth, min_node

stopping rules standing in for n_min.

newdata

optional points to predict.

seed

seed for the bootstrap and per-node draws.

Value

list: prediction, oob_prediction, oob_mse, train_mse, mtry, mtry_rule, subset_drawn_per, n_oob_missing, B, n, p, method.

Details

The variable subset is drawn per NODE, not per tree. Bagging already averages B identically distributed trees, so the bias is unchanged and the only gain is variance reduction; averaging B i.d. variables with pairwise correlation rho leaves rho sigma^2 behind, so the gain is bounded by how decorrelated the trees are. The per-node draw is where that decorrelation comes from.

The default mtry is floor(p/3) for regression. The classification default floor(sqrt(p)) is a different number – they cross at p = 9, and above it the regression rule is the larger of the two.

Out-of-bag error is free: each observation is absent from about 36.8% of the trees (7.55), and averaging only those gives a nearly-cross-validated error with no extra fitting.

References

Hastie, Tibshirani and Friedman (2009), Ch. 15, Algorithm 15.1 and Secs. 15.2-15.3; Breiman (2001).

Examples

set.seed(1)
X <- matrix(stats::rnorm(300), 100)
morie_esl_random_forest(X, X[, 1] + stats::rnorm(100), B = 10)$mtry
#> [1] 1