LoRA forward with alpha/r scaling (Kamath Ch 4)

h = W0 x + (alpha/r) B A x. km058's Eq 4.5 has no scaling; this named module carries the implementation's alpha/r, so the two stay separate.

Usage

morie_kamath_lora_weight_update(W0, A, B, alpha, r, x)

Arguments

W0

Frozen weight.

A

(r, k).

B

(d, r).

alpha

Scale numerator.

r

Rank.

x

Input.

Value

List with h, base, delta, scaling, n_trainable.

Examples

morie_kamath_lora_weight_update(diag(2), matrix(c(1, 1), 1),
    matrix(c(2, 0), 2), 4, 1, c(1, 3))
#> $h
#> [1] 33  3
#> 
#> $base
#> [1] 1 3
#> 
#> $delta
#> [1] 32  0
#> 
#> $estimate
#> [1] 33
#> 
#> $scaling
#> [1] 4
#> 
#> $rank
#> [1] 1
#> 
#> $alpha
#> [1] 4
#> 
#> $n_trainable
#> [1] 4
#> 
#> $n_frozen
#> [1] 4
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "LoRA forward h = W0 x + (alpha/r) B A x"
#>