Autoregressive next-token cross-entropy (Geron Ch 16, morie.fn grgptl)

Autoregressive next-token cross-entropy (Geron Ch 16, morie.fn grgptl)

Usage

morie_geron_gpt_autoregressive_loss(logits, targets, reduction = "sum")

Arguments

logits

(T, V) matrix.

targets

0-based token ids, length T.

reduction

"sum" or "mean"; selects estimate only.

Value

List with loss, mean_loss, perplexity, per_token_loss, target_logprob.

Examples

morie_geron_gpt_autoregressive_loss(logits = c(1, 2, 3, 4, 5, 6, 7, 8), targets = 5L)
#> $loss
#> [1] 2.45834
#> 
#> $mean_loss
#> [1] 2.45834
#> 
#> $perplexity
#> [1] 11.68539
#> 
#> $per_token_loss
#> [1] 2.45834
#> 
#> $target_logprob
#> [1] -2.45834
#> 
#> $estimate
#> [1] 2.45834
#> 
#> $n
#> [1] 1
#>