VeRA forward pass (Kamath Eq 4.10)

h = W_0 x + Lambda_b B Lambda_d A x; only the two diagonals train.

Usage

morie_kamath_ch4_vera_forward(W_0, Lambda_b, Lambda_d, A, B, x)

Arguments

W_0

Frozen weight.

Lambda_b, Lambda_d

Diagonals.

A, B

Frozen shared factors.

x

Input.

Value

List with h, delta_h, r, n_trainable.

Examples

morie_kamath_ch4_vera_forward(diag(2), c(2, 3), c(5), matrix(c(1,
    1), 1), matrix(c(1, 2), 2), c(1, 2))
#> $h
#> [1] 31 92
#> 
#> $base
#> [1] 1 2
#> 
#> $delta_h
#> [1] 30 90
#> 
#> $r
#> [1] 1
#> 
#> $n_trainable
#> [1] 3
#> 
#> $n_trainable_lora
#> [1] 4
#> 
#> $estimate
#> [1] 31
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "VeRA forward pass (Kamath Eq 4.10)"
#>