Prefix tuning key/value concatenation (Kamath Ch 4)

[P_K; K] and [P_V; V]; with Q the attention is run through morie_alammar_sdp_attention.

Usage

morie_kamath_prefix_tuning(prefix_K, prefix_V, K_input, V_input, Q = NULL)

Arguments

prefix_K, prefix_V

Virtual key/value tokens.

K_input, V_input

Real keys and values.

Q

Optional queries.

Value

List with K, V, prefix_len, n_trainable.

Examples

morie_kamath_prefix_tuning(prefix_K = c(1, 2, 3, 4, 5, 6, 7, 8),
  prefix_V = c(1, 2, 3, 4, 5, 6, 7, 8), K_input = c(1, 2, 3, 4, 5, 6, 7, 8),
  V_input = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $K
#>       [,1]
#>  [1,]    1
#>  [2,]    2
#>  [3,]    3
#>  [4,]    4
#>  [5,]    5
#>  [6,]    6
#>  [7,]    7
#>  [8,]    8
#>  [9,]    1
#> [10,]    2
#> [11,]    3
#> [12,]    4
#> [13,]    5
#> [14,]    6
#> [15,]    7
#> [16,]    8
#> 
#> $V
#>       [,1]
#>  [1,]    1
#>  [2,]    2
#>  [3,]    3
#>  [4,]    4
#>  [5,]    5
#>  [6,]    6
#>  [7,]    7
#>  [8,]    8
#>  [9,]    1
#> [10,]    2
#> [11,]    3
#> [12,]    4
#> [13,]    5
#> [14,]    6
#> [15,]    7
#> [16,]    8
#> 
#> $prefix_len
#> [1] 8
#> 
#> $seq_len
#> [1] 16
#> 
#> $n_trainable
#> [1] 16
#> 
#> $estimate
#> [1] 8
#> 
#> $n
#> [1] 16
#> 
#> $method
#> [1] "Prefix tuning key/value concatenation"
#>