Auxiliary-task pretraining then fine-tuning (Geron Ch 11, hmauxpt) — morie_geron_auxiliary_task

Gradient descent on the auxiliary task, then the SAME parameter vector is fine-tuned on the target; the from-scratch control gets an identical budget, so transfer_gain is measured, not asserted.

Usage

morie_geron_auxiliary_task_pretraining(
  model,
  aux_data,
  target_data,
  aux_epochs = 200,
  epochs = 20,
  lr = 0.05
)

Arguments

model: Initial parameters or NULL for zeros.
aux_data, target_data: Lists of (X, y).
aux_epochs, epochs: Gradient steps.
lr: Step size.

Value

List with theta, theta_pretrained, theta_scratch, target_loss, scratch_loss, transfer_gain, aux_losses, finetune_losses, scratch_losses.

Examples

Xa <- matrix(c(1, 0, 0, 1, 1, 1, 2, 1), 4, 2, byrow = TRUE)
Xt <- matrix(c(1, 1, 2, 0), 2, 2, byrow = TRUE)
morie_geron_auxiliary_task_pretraining(NULL, list(Xa, c(1, 2,
    3, 4)), list(Xt, c(3, 2)), aux_epochs = 50, epochs = 5)
#> $theta
#> [1] 1.067845 1.795500
#> 
#> $theta_pretrained
#> [1] 1.144467 1.766182
#> 
#> $theta_scratch
#> [1] 1.018965 0.552655
#> 
#> $target_loss
#> [1] 0.01854313
#> 
#> $scratch_loss
#> [1] 1.020854
#> 
#> $transfer_gain
#> [1] 1.002311
#> 
#> $aux_losses
#>  [1] 7.50000000 4.96171875 3.31571631 2.24636731 1.54980412 1.09433896
#>  [7] 0.79490250 0.59653493 0.46372368 0.37351712 0.31107697 0.26680582
#> [13] 0.23449185 0.21011059 0.19105095 0.17561538 0.16269695 0.15157077
#> [19] 0.14175928 0.13294516 0.12491509 0.11752342 0.11066860 0.10427803
#> [25] 0.09829817 0.09268819 0.08741580 0.08245459 0.07778223 0.07337933
#> [31] 0.06922869 0.06531480 0.06162345 0.05814153 0.05485689 0.05175815
#> [37] 0.04883467 0.04607646 0.04347413 0.04101883 0.03870224 0.03651651
#> [43] 0.03445423 0.03250843 0.03067253 0.02894031 0.02730593 0.02576384
#> [49] 0.02430885 0.02293602
#> 
#> $finetune_losses
#> [1] 0.04573333 0.03479406 0.02809310 0.02375716 0.02076130
#> 
#> $scratch_losses
#> [1] 6.500000 3.970000 2.559250 1.760820 1.298288
#> 
#> $estimate
#> [1] 0.01854313
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "Auxiliary-task pretraining then fine-tuning, against an equal-budget scratch control"
#>