Attention with linear biases: the pre-softmax score of query i and
key j is penalised by slope m times the distance between i and j, and
no position embeddings are used anywhere. Head k of n receives the
geometric slope 2^(-8k/n). Pass slopes for explicit per-head
slopes. There is exactly one implementation: this function delegates
to Atalib.
Usage
Paligi(y = NULL, Q = NULL, K = NULL, V = NULL, slopes = NULL, causal = FALSE)
Arguments
- y
Optional scores input, see Atalib.
- Q
Query matrix.
- K
Key matrix.
- V
Value matrix.
- slopes
Optional explicit head slope or slopes.
- causal
Mask keys after the query position. Default FALSE.
Value
List as returned by Atalib.
References
Press, O., Smith, N. A. and Lewis, M. (2022), Train Short, Test Long:
Attention with Linear Biases Enables Input Length Extrapolation, ICLR 2022,
arXiv:2108.12409, page 4 and Section 3.
Examples
Paligi(Q = diag(2), K = diag(2), V = diag(2))$estimate
#> [1] 0.670625
