AdaLoRA SVD update with importance-based pruning (Kamath Ch 4) — morie_kamath_adalora_rank_allocation • rmorie

P diag(s) Q^T after zeroing all but the top-importance singular values. kept is 0-BASED and sorted.

Usage

morie_kamath_adalora_rank_allocation(
  P,
  s,
  Q,
  importance = NULL,
  target_rank = NULL
)

Arguments

P: Left factor (rows x r).
s: Singular values.
Q: Right factor (rows x r).
importance: Optional scores.
target_rank: How many triplets to keep.

Value

List with estimate, Delta_W, s_pruned, kept.

Examples

morie_kamath_adalora_rank_allocation(P = 0.5, s = 5L, Q = 0.5)
#> $estimate
#> [1] 1.25
#> 
#> $Delta_W
#>      [,1]
#> [1,] 1.25
#> 
#> $s_pruned
#> [1] 5
#> 
#> $kept
#> [1] 0
#> 
#> $target_rank
#> [1] 1
#> 
#> $effective_rank
#> [1] 1
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "AdaLoRA SVD update with importance-based pruning"
#>