Chinchilla compute-optimal split (Kamath Ch 1)

N = sqrt(C / (6 * 20)), D = 20 N, checked against C = 6 N D.

Usage

morie_kamath_chinchilla_compute_optimal(
  compute_budget,
  alpha = 0.5,
  beta = 0.5,
  tokens_per_param = 20,
  flops_per_token_param = 6
)

Arguments

compute_budget

Total FLOPs.

alpha, beta

Exponents summing to 1.

tokens_per_param

Tokens per parameter.

flops_per_token_param

FLOPs constant.

Value

List with estimate, N_opt, D_opt, compute_check.

Examples

morie_kamath_chinchilla_compute_optimal(compute_budget = 5L)
#> $estimate
#> [1] 0.2041241
#> 
#> $N_opt
#> [1] 0.2041241
#> 
#> $D_opt
#> [1] 4.082483
#> 
#> $tokens_per_param
#> [1] 20
#> 
#> $compute_budget
#> [1] 5
#> 
#> $compute_check
#> [1] 5
#> 
#> $alpha
#> [1] 0.5
#> 
#> $beta
#> [1] 0.5
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "Chinchilla compute-optimal split (Kamath Ch 1)"
#>