Formula: K reference points per query; sparse attention
Usage
Defdtr(x, queries, K = 4, offsets = NULL, weights = NULL, seed = 42)
Arguments
- x
An H x W feature map.
- queries
A Q x 2 matrix of reference points in [0, 1].
- K
Sampling points per query.
- offsets
Q x K x 2 offsets in pixels, or NULL.
- weights
Q x K attention weights, or NULL for 1/K each.
- seed
Seed of the deterministic stream.
Value
List with estimate, out, samples,
ref_pixels, Q, K, method.
Details
Each query attends to only K sampled locations around its reference
point instead of every position, so the cost is O(K) rather than
O(HW) and convergence no longer needs 500 epochs. The samples are
read by bilinear interpolation, which is what makes the learned
offsets differentiable. With zero offsets and equal weights the
output is exactly the feature at the reference point.
References
Zhu et al. (2021), Deformable DETR, ICLR 2021.
Examples
x <- outer(1:16, 1:16, function(i, j) sin(i / 3) + cos(j / 4))
q <- rbind(c(0.25, 0.25), c(0.75, 0.5))
d <- Defdtr(x, q, K = 4)
d$out
#> [1] 1.3773769 -0.5198049
