VeRA adapter (Kamath Ch 4)

h = W0 x + Lambda_b B Lambda_d A x; only the two diagonals train, A and B stay frozen and shared.

Usage

morie_kamath_vera_adapter(W0, A_frozen, B_frozen, lam_b, lam_d, x)

Arguments

W0

Frozen weight.

A_frozen, B_frozen

Shared factors.

lam_b, lam_d

Trainable diagonals.

x

Input.

Value

List with h, delta, rank, n_trainable.

Examples

morie_kamath_vera_adapter(diag(2), matrix(c(1, 1), 1), matrix(c(1,
    2), 2), c(2, 3), c(5), c(1, 2))
#> $h
#> [1] 31 92
#> 
#> $base
#> [1] 1 2
#> 
#> $delta
#> [1] 30 90
#> 
#> $estimate
#> [1] 31
#> 
#> $rank
#> [1] 1
#> 
#> $n_trainable
#> [1] 3
#> 
#> $n_trainable_lora_equivalent
#> [1] 4
#> 
#> $n_frozen
#> [1] 8
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "VeRA h = W0 x + Lambda_b B Lambda_d A x"
#>