LoRA forward pass (Kamath Eq 4.5)

h = W_0 x + B A x; dW = BA has rank at most r, and that rank is reported.

Usage

morie_kamath_ch4_lora_forward(W_0, B, A, x)

Arguments

W_0

Frozen (d, k) weight.

B

(d, r).

A

(r, k).

x

Input vector of length k.

Value

List with h, base, delta_h, r, delta_W_rank.

Examples

morie_kamath_ch4_lora_forward(diag(2), matrix(c(1, 0), 2), matrix(c(1,
    0), 1), c(1, 2))
#> $h
#> [1] 2 2
#> 
#> $base
#> [1] 1 2
#> 
#> $delta_h
#> [1] 1 0
#> 
#> $r
#> [1] 1
#> 
#> $delta_W_rank
#> [1] 1
#> 
#> $estimate
#> [1] 2
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "LoRA forward pass (Kamath Eq 4.5)"
#>