Houlsby bottleneck adapter (Kamath Ch 4)

h + W_up GELU(W_down h) with m << d enforced. approximate = "none" is the exact erf GELU (R: z * pnorm(z)), "tanh" the tanh approximation.

Usage

morie_kamath_houlsby_adapter(h, W_down, W_up, approximate = "none")

Arguments

h

Hidden rows.

W_down

(m, d).

W_up

(d, m).

approximate

"none" or "tanh".

Value

List with estimate, h_adapted, bottleneck.

Examples

morie_kamath_houlsby_adapter(matrix(c(1, 2, 3), 1), matrix(c(1,
    0, 0), 1), matrix(c(1, 0, 0), 3))
#> $estimate
#> [1] 0.8413447
#> 
#> $h_adapted
#>          [,1] [,2] [,3]
#> [1,] 1.841345    2    3
#> 
#> $bottleneck
#>           [,1]
#> [1,] 0.8413447
#> 
#> $m
#> [1] 1
#> 
#> $d
#> [1] 3
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Houlsby bottleneck adapter (Kamath Ch 4)"
#>