GATv2 attention layer

GATv2 moves the nonlinearity inside: the score is a linear read-out OF a LeakyReLU, not a LeakyReLU of a linear read-out. That single reordering is what turns static attention into dynamic attention. With W = I and a = 1 the score becomes sum_c LeakyReLU(h_ic) + sum_c LeakyReLU(h_jc), which differs from GAT's LeakyReLU(sum h_i + sum h_j) whenever a feature vector mixes signs.

Usage

GatV2(A, X)

Arguments

A

Square adjacency matrix.

X

Node feature matrix, one row per node.

Value

List with estimate, H, alpha_first, n, method.

Details

Formula: e_ij = sum_c LeakyReLU(h_ic) + sum_c LeakyReLU(h_jc); alpha = softmax over the neighbourhood; h_i' = sigmoid(sum alpha h_j).

References

Brody, Alon and Yahav (2022), How attentive are graph attention networks?, ICLR 2022, eq. (7). arXiv:2105.14491

Examples

set.seed(1)
A <- matrix(c(0, 1, 0, 1, 0, 1, 0, 1, 0), 3, 3, byrow = TRUE)
X <- matrix(rnorm(6), 3, 2)
GatV2(A, X)
#> $estimate
#> [1] 0.544682
#> 
#> $H
#>           [,1]      [,2]
#> [1,] 0.4009178 0.7762620
#> [2,] 0.3898620 0.7357514
#> [3,] 0.4693472 0.4959520
#> 
#> $alpha_first
#> [1] 0.7224885 0.2775115
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "GATv2 eq. (7): a' LeakyReLU(W[h_i || h_j]) with W = I, a = 1"
#> 
#> attr(,"class")
#> [1] "morie_rich_result" "list"