Entropy-based attention regularisation (Kamath Eq 6.21)

-lam times the summed per-layer MEAN attention row entropy. Rows must sum to 1; 0 log 0 is taken as 0.

Usage

morie_kamath_ch6_ear_entropy_reg(A, L = NULL, lam = 1)

Arguments

A

List of attention matrices.

L

Optional layer count.

lam

Non-negative weight.

Value

List with estimate, per_layer_entropy, total_entropy.

Examples

morie_kamath_ch6_ear_entropy_reg(list(matrix(c(0.5, 1, 0.5, 0),
    2)), lam = 2)
#> $estimate
#> [1] -0.6931472
#> 
#> $per_layer_entropy
#> [1] 0.3465736
#> 
#> $total_entropy
#> [1] 0.3465736
#> 
#> $lam
#> [1] 2
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "entropy-based attention regularisation (Kamath Eq 6.21)"
#>