Transformer architecture (Vaswani et al. 2017)

Multi-head attention (heads via morie_geron_scaled_dot_product), residual + layer norm, position-wise feedforward, residual + layer norm.

Usage

morie_geron_transformer(
  X,
  n_heads = 2,
  d_model = NULL,
  n_layers = 2,
  d_ff = NULL,
  seed = 0,
  mask = NULL
)

Arguments

X

Input sequence (T, d_model).

n_heads, d_model, n_layers, d_ff, seed, mask

As in Python original.

Value

list with Y, attention, total_params, d_model, d_ff, n_heads, n_layers, estimate, n, method.

Examples

X <- matrix(c(1, 0, 0.5, 0.2, 1, 0, 0.3, 0.7), nrow = 4)   # four positions, two dimensions
morie_geron_transformer(X, n_heads = 2, n_layers = 1)$total_params
#> [1] 66