Transfer learning: reuse pretrained model, fine-tune on new task

The first n_frozen weight matrices are used forward-only and never updated; the trainable tail is optimised by backprop through a tanh stack (final layer linear).

Usage

morie_geron_transfer_learning(
  pretrained_model,
  X,
  y,
  n_frozen = 1,
  epochs = 200,
  lr = 0.05
)

Arguments

pretrained_model

List of weight matrices W_0 (d_in,h1), W_1 (h1,h2), ...

X

New-task inputs (n, d_in).

y

New-task targets (n,) or (n, d_out).

n_frozen

Leading layers to freeze.

epochs, lr

As in Python original.

Value

list with weights, frozen, loss_curve, initial_loss, final_loss, trainable_params, total_params, estimate, n, method.

Examples

morie_geron_transfer_learning(pretrained_model = c(1, 2, 3, 4, 5, 6, 7, 8),
  X = c(1, 2, 3, 4, 5, 6, 7, 8), y = c(1, 2, 3, 4, 5, 6, 7, 8))
#> $weights
#> $weights[[1]]
#>      [,1]
#> [1,]    1
#> 
#> $weights[[2]]
#>      [,1]
#> [1,]    2
#> 
#> $weights[[3]]
#>      [,1]
#> [1,]    3
#> 
#> $weights[[4]]
#>      [,1]
#> [1,]    4
#> 
#> $weights[[5]]
#>      [,1]
#> [1,]    5
#> 
#> $weights[[6]]
#>      [,1]
#> [1,]    6
#> 
#> $weights[[7]]
#>          [,1]
#> [1,] 6.999926
#> 
#> $weights[[8]]
#>          [,1]
#> [1,] 4.500007
#> 
#> 
#> $frozen
#> [1] 1
#> 
#> $loss_curve
#>   [1] 17.499907 15.172432 13.287176 11.760117 10.523199  9.521294  8.709751
#>   [8]  8.052400  7.519946  7.088658  6.739314  6.456345  6.227140  6.041484
#>  [15]  5.891103  5.769293  5.670628  5.590709  5.525974  5.473539  5.431067
#>  [22]  5.396664  5.368798  5.346227  5.327944  5.313134  5.301139  5.291423
#>  [29]  5.283552  5.277177  5.272014  5.267831  5.264443  5.261699  5.259476
#>  [36]  5.257676  5.256217  5.255036  5.254079  5.253304  5.252676  5.252168
#>  [43]  5.251756  5.251422  5.251152  5.250933  5.250756  5.250612  5.250496
#>  [50]  5.250402  5.250325  5.250264  5.250213  5.250173  5.250140  5.250113
#>  [57]  5.250092  5.250074  5.250060  5.250049  5.250040  5.250032  5.250026
#>  [64]  5.250021  5.250017  5.250014  5.250011  5.250009  5.250007  5.250006
#>  [71]  5.250005  5.250004  5.250003  5.250003  5.250002  5.250002  5.250001
#>  [78]  5.250001  5.250001  5.250001  5.250001  5.250000  5.250000  5.250000
#>  [85]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#>  [92]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#>  [99]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [106]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [113]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [120]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [127]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [134]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [141]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [148]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [155]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [162]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [169]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [176]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [183]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [190]  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000  5.250000
#> [197]  5.250000  5.250000  5.250000  5.250000  5.250000
#> 
#> $initial_loss
#> [1] 17.49991
#> 
#> $final_loss
#> [1] 5.25
#> 
#> $trainable_params
#> [1] 7
#> 
#> $total_params
#> [1] 8
#> 
#> $estimate
#> [1] 5.25
#> 
#> $n
#> [1] 8
#> 
#> $method
#> [1] "Fine-tuning the unfrozen tail by backpropagation; frozen layers forward-only"
#>