Two-stream attention masks induced by a factorization order

Returns the pair of logical matrices that make XLNet's permutation objective computable in a single pass over an unpermuted sequence. content[i, j] is TRUE when position i may attend to position j.

Usage

morie_permutation_attention_masks(permutation)

Arguments

permutation

Integer permutation of 0:(T-1), where permutation[t] is the sequence position generated t-th.

Value

A list with content, query and rank.

Details

The content stream sees \(z_{\le t}\) – itself included – and the query stream sees only \(z_{<t}\). The single difference is the diagonal, and that diagonal is the whole reason two streams exist: a query able to see its own token would make predicting that token trivial, while a content stream unable to see it could not encode it for later positions.

References

Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNet. arXiv:1906.08237, eq (3).

Examples

morie_permutation_attention_masks(c(2L, 0L, 3L, 1L))
#> $content
#>       [,1]  [,2] [,3]  [,4]
#> [1,]  TRUE FALSE TRUE FALSE
#> [2,]  TRUE  TRUE TRUE  TRUE
#> [3,] FALSE FALSE TRUE FALSE
#> [4,]  TRUE FALSE TRUE  TRUE
#> 
#> $query
#>       [,1]  [,2]  [,3]  [,4]
#> [1,] FALSE FALSE  TRUE FALSE
#> [2,]  TRUE FALSE  TRUE  TRUE
#> [3,] FALSE FALSE FALSE FALSE
#> [4,]  TRUE FALSE  TRUE FALSE
#> 
#> $rank
#> [1] 1 3 0 2
#>