Fine-tuning via TRL (Transformer Reinforcement Learning) library

"sft" delegates to morie_geron_sft. "dpo": Direct Preference Optimisation, loss = -log sigmoid(beta*(Delta_policy - Delta_ref)). "ppo": clipped surrogate min(rA, clip(r,1-eps,1+eps)A).

Usage

morie_geron_trl_finetune(
  model = NULL,
  dataset,
  method = "sft",
  epochs = 200,
  lr = 0.1,
  beta = 0.1,
  clip_eps = 0.2,
  theta_ref = NULL
)

Arguments

model

Optional initial parameters; zeros by default.

dataset

Shape depends on method (see Python docstring).

method

"sft", "dpo", or "ppo".

epochs, lr, beta, clip_eps, theta_ref

As in Python original.

Value

list (shape depends on method) with theta, loss, loss_curve, ..., estimate, n, method.

Examples

morie_geron_trl_finetune(dataset = data.frame(x = c(1, 2, 3, 4), y = c(2, 4, 5, 9)))
#> $theta
#>            [,1]      [,2]
#> [1,] -0.2098088 0.2098088
#> 
#> $loss
#> [1] 0.6419534
#> 
#> $loss_curve
#>   [1] 0.6931472 0.6814282 0.6724520 0.6655680 0.6602784 0.6562047 0.6530597
#>   [8] 0.6506257 0.6487374 0.6472692 0.6461250 0.6452317 0.6445328 0.6439853
#>  [15] 0.6435556 0.6432179 0.6429522 0.6427429 0.6425779 0.6424476 0.6423448
#>  [22] 0.6422634 0.6421991 0.6421482 0.6421079 0.6420760 0.6420507 0.6420306
#>  [29] 0.6420147 0.6420021 0.6419921 0.6419841 0.6419778 0.6419728 0.6419688
#>  [36] 0.6419656 0.6419631 0.6419611 0.6419596 0.6419583 0.6419573 0.6419565
#>  [43] 0.6419559 0.6419554 0.6419550 0.6419546 0.6419544 0.6419542 0.6419540
#>  [50] 0.6419539 0.6419538 0.6419537 0.6419537 0.6419536 0.6419536 0.6419535
#>  [57] 0.6419535 0.6419535 0.6419535 0.6419535 0.6419534 0.6419534 0.6419534
#>  [64] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#>  [71] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#>  [78] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#>  [85] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#>  [92] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#>  [99] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [106] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [113] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [120] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [127] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [134] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [141] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [148] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [155] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [162] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [169] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [176] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [183] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [190] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> [197] 0.6419534 0.6419534 0.6419534 0.6419534 0.6419534
#> 
#> $accuracy
#> [1] 0.5
#> 
#> $trainer
#> [1] "sft"
#> 
#> $estimate
#> [1] 0.6419534
#> 
#> $n
#> [1] 2
#> 
#> $method
#> [1] "TRL SFT trainer (delegated to hmsft)"
#>