Bottleneck adapters (Kamath Eq 4.1-4.2)

Usage

morie_kamath_ch4_series_adapter(H_o, W_down, W_up, f = NULL)

morie_kamath_ch4_parallel_adapter(H_o, H_i, W_down, W_up, f = NULL)

Arguments

H_o: Layer output.
W_down, W_up: Bottleneck factors.
f: Activation; default ReLU.
H_i: Layer input feeding the parallel branch.

Value

List with output, delta, bottleneck_rank.

Details

One core serves both Python modules exactly as km055 imports km054's _adapter_core: km054 -> morie_kamath_ch4_series_adapter (bottleneck fed by H_o), km055 -> morie_kamath_ch4_parallel_adapter (fed by the layer input H_i). Passing H_i = H_o reproduces the series adapter exactly.

Examples

morie_kamath_ch4_series_adapter(matrix(c(1, 2), 1), matrix(c(1,
    0), 2), matrix(c(1, 1), 1))
#> $output
#>      [,1] [,2]
#> [1,]    2    3
#> 
#> $delta
#>      [,1] [,2]
#> [1,]    1    1
#> 
#> $bottleneck_rank
#> [1] 1
#> 
#> $estimate
#> [1] 2
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "series adapter (Kamath Eq 4.1)"
#>