Fine-tune a frozen encoder with a softmax head (Geron Ch 15, hmbftn)

bert is any function(X) -> (n, d) pooled embeddings. The head is trained by full-batch gradient descent with the exact gradient X^T (p - onehot) / n. Labels and predictions are 0-based.

Usage

morie_geron_bert_finetune(bert, X, y, epochs = 100, lr = 0.1, l2 = 0)

Arguments

bert: Embedding function.
X: Inputs passed to bert.
y: 0-based class labels.
epochs: Steps.
lr: Step size.
l2: Non-negative penalty on W (bias unpenalised).

Value

List with W, b, losses, accuracy, predict, probabilities, embeddings.

Examples

ident <- function(A) as.matrix(A)
morie_geron_bert_finetune(ident, matrix(c(1, 0, 0, 1, 1, 1),
    3, 2, byrow = TRUE), c(0, 1, 0), epochs = 25, lr = 0.5, l2 = 0.01)
#> $W
#>            [,1]       [,2]
#> [1,]  1.4507439 -1.4507439
#> [2,] -0.5863562  0.5863562
#> 
#> $b
#> [1] -0.004884105  0.004884105
#> 
#> $losses
#>  [1] 0.6931472 0.5763120 0.5125557 0.4671385 0.4306377 0.3998937 0.3734852
#>  [8] 0.3505702 0.3305514 0.3129658 0.2974405 0.2836705 0.2714038 0.2604315
#> [15] 0.2505790 0.2417001 0.2336716 0.2263891 0.2197638 0.2137197 0.2081917
#> [22] 0.2031235 0.1984664 0.1941779 0.1902209
#> 
#> $accuracy
#> [1] 1
#> 
#> $predict
#> function (Xnew) 
#> {
#>     Zn <- bert(Xnew)
#>     Zn <- if (is.matrix(Zn)) 
#>         Zn
#>     else matrix(as.numeric(Zn), nrow = 1)
#>     .morie_gr_need(ncol(Zn) == dim_, "predict: encoder width mismatch")
#>     apply(Zn %*% W + matrix(b, nrow(Zn), K, byrow = TRUE), 1, 
#>         which.max) - 1L
#> }
#> <environment: 0x5621544dba98>
#> 
#> $probabilities
#>           [,1]       [,2]
#> [1,] 0.9474356 0.05256441
#> [2,] 0.2346065 0.76539354
#> [3,] 0.8480009 0.15199909
#> 
#> $embeddings
#>      [,1] [,2]
#> [1,]    1    0
#> [2,]    0    1
#> [3,]    1    1
#> 
#> $estimate
#> [1] 0.1902209
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "Softmax classification head fine-tuned on pooled [CLS] embeddings"
#>