Liu et al. (2021), "Swin Transformer", ICCV 2021, arXiv:2103.14030,
Eq 4: Attention(Q,K,V) = SoftMax(Q K' / sqrt(d) + B) V inside each
non-overlapping M x M window, with B (M^2 x M^2) looked up from the
parameterised (2M-1) x (2M-1) table by the tokens' relative
position along each axis.
Usage
Swinmw(x, window_size, relative_bias = NULL, WQ = NULL, WK = NULL, WV = NULL)
Arguments
- x
Feature map, array with dim c(H, W, d); M must divide H, W.
- window_size
Window side M.
- relative_bias
Optional (2M-1) x (2M-1) table; zeros if
omitted.
- WQ, WK, WV
Optional projections (d x d_k / d x d_k / d x d_v);
identity if omitted.
Value
List with output (H x W x d_v array), bias,
n_windows, tokens_per_window, estimate,
n, method.
References
Liu, Z. et al. (2021), ICCV 2021, arXiv:2103.14030, Section 3.2, Eq 4.
Examples
set.seed(6)
x <- array(rnorm(4 * 4 * 2), c(4, 4, 2))
r <- Swinmw(x, window_size = 2)
c(r$n_windows, r$tokens_per_window)
#> [1] 4 4
