Native SGD fine-tuning with frozen params, warmup and weight decay (hmfth)

Native SGD fine-tuning with frozen params, warmup and weight decay (hmfth)

Usage

morie_geron_finetune_lm(
  model,
  dataset,
  epochs = 10,
  lr = 0.01,
  theta = NULL,
  freeze = NULL,
  batch_size = NULL,
  weight_decay = 0,
  warmup = 0
)

Arguments

model

function(theta, batch) -> list(loss, grad).

dataset

Training examples.

epochs, lr, theta, freeze, batch_size, weight_decay, warmup

Config.

Value

List with theta, theta_init, loss_history, drift, lr_schedule, frozen.

Examples

task <- function(th, batch) list((th[1] - 3)^2, 2 * (th[1] -
    3))
morie_geron_finetune_lm(task, list(1), epochs = 1, lr = 0.1,
    theta = 0)
#> $theta
#> [1] 0.6
#> 
#> $theta_init
#> [1] 0
#> 
#> $loss_history
#> [1] 9
#> 
#> $grad_norms
#> [1] 6
#> 
#> $lr_schedule
#> [1] 0.1
#> 
#> $drift
#> [1] 0.6
#> 
#> $n_steps
#> [1] 1
#> 
#> $frozen
#> [1] FALSE
#> 
#> $n_frozen
#> [1] 0
#> 
#> $weight_decay
#> [1] 0
#> 
#> $warmup
#> [1] 0
#> 
#> $estimate
#> [1] 9
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "native SGD fine-tuning loop with frozen parameters, warmup and weight decay"
#>