The wide part is a generalised linear model on the raw wide features and their cross-product transformations; the deep part is a feed-forward network on the dense features. Their output log-odds are summed and fed to one logistic loss, so the two parts are trained jointly rather than ensembled: every parameter, in both parts and in the shared bias, sees the same gradient step.
Usage
WideD(
X_wide,
X_deep,
y,
hidden = 8L,
epochs = 300L,
lr = 0.05,
seed = 1L,
crosses = NULL,
l2 = 0
)Arguments
- X_wide
Numeric matrix of wide features, \(n\) rows.
- X_deep
Numeric matrix of dense features for the network.
- y
Binary labels in {0, 1}, length \(n\).
Integer vector of hidden layer widths. Default 8.
- epochs
Number of full-batch gradient steps.
- lr
Step size.
- seed
Seed of the deterministic generator used to initialise weights.
- crosses
List of length-2 integer vectors giving 0-based index pairs into the columns of
X_wide; each adds the eq. (1) product of those columns to the wide design.- l2
Ridge penalty on every weight (not on biases). Default 0.
Value
Named list with coef_wide, coef_deep,
bias, hidden_weights, hidden_bias,
fitted, loss, n, n_wide, n_deep,
epochs, method.
Details
In the numbering of Cheng et al. (2016): eq. (1) is the cross-product transformation \(\phi_k(x) = \prod_i x_i^{c_{ki}}\), which lets the linear wide part express interactions; eq. (2) is the hidden-layer computation \(a^{(l+1)} = f(W^{(l)} a^{(l)} + b^{(l)})\) with \(f\) the rectifier; and eq. (3) is the prediction $$P(Y = 1 | x) = \sigma(w_{wide}^T [x, \phi(x)] + w_{deep}^T a^{(l_f)} + b).$$
The paper trains with mini-batch stochastic gradient descent, FTRL with L1 for the wide part and AdaGrad for the deep part. This implementation instead uses full-batch gradient descent with a fixed step size for a fixed number of epochs, and initialises from a reproducible linear congruential generator. The model, eq. (1) to (3), and the joint logistic objective are the paper's; only the optimiser is not, and it is replaced so that the fit is a deterministic function of the inputs. Do not read the result as reproducing the paper's benchmark numbers, which depend on that optimiser and on embeddings this function does not fit.
Mirrors morie.fn.wideD on the Python side.