iTransformer: variates as tokens, attention across variates

Quoted from the paper: (1) "h^0_n = Embedding(X_{:,n}); H^{l+1} = TrmBlock(H^l); Yhat_{:,n} = Projection(h^L_n)"; (2) "LayerNorm(H) = {[h_n - Mean(h_n)]/sqrt(Var(h_n)) | n = 1..N}"; attention scores "A_{i,j} = (Q K^T / sqrt(d_k))_{i,j}" – Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L. and Long, M., iTransformer, ICLR 2024 (arXiv:2310.06625). The inversion is the point: each VARIATE series becomes one token, so attention is N x N over variates rather than T x T over time.

Usage

morie_itrans(
  x,
  wembed,
  bembed,
  wq,
  wk,
  wv,
  wffn1,
  bffn1,
  wffn2,
  bffn2,
  wproj,
  bproj
)

Arguments

x

list of N variate series, each of length T

wembed, bembed

the embedding map

wq, wk, wv

the attention projections

wffn1, bffn1, wffn2, bffn2

the feed-forward network

wproj, bproj

the output projection

Value

list(forecast, attn, tokens, nvariates, T, D, horizon, attndiag, mean, first, sumsq)

Examples

set.seed(6)
x <- list(rnorm(8), rnorm(8))
r <- morie_itrans(x,
                  wembed = matrix(rnorm(32, 0, 0.3), 4, 8), bembed = rep(0, 4),
                  wq = matrix(rnorm(8, 0, 0.4), 2, 4),
                  wk = matrix(rnorm(8, 0, 0.4), 2, 4),
                  wv = matrix(rnorm(16, 0, 0.4), 4, 4),
                  wffn1 = matrix(rnorm(16, 0, 0.4), 4, 4), bffn1 = rep(0, 4),
                  wffn2 = matrix(rnorm(16, 0, 0.4), 4, 4), bffn2 = rep(0, 4),
                  wproj = matrix(rnorm(8, 0, 0.4), 2, 4), bproj = rep(0, 2))
str(r, max.level = 1)
#> List of 11
#>  $ forecast :List of 2
#>  $ attn     :List of 2
#>  $ tokens   :List of 2
#>  $ nvariates: int 2
#>  $ T        : int 8
#>  $ D        : int 4
#>  $ horizon  : int 2
#>  $ attndiag : num 0.514
#>  $ mean     : num -0.125
#>  $ first    : num -0.928
#>  $ sumsq    : num 1.04