Instruction-tuning cross entropy (Kamath Ch 4)

Causal-LM cross entropy with the instruction masked out (-100).

Usage

morie_kamath_instruction_tuning_loss(logits, response_mask, targets)

Arguments

logits

(T, V) logits.

response_mask

Logical or 0/1 mask.

targets

Token ids (0-BASED).

Value

List with estimate, perplexity, n_response_tokens.

Examples

morie_kamath_instruction_tuning_loss(matrix(c(1, 0, 2, 0, 1,
    0), 3), c(0, 1, 1), c(0, 1, 0))
#> $estimate
#> [1] 0.2200948
#> 
#> $loss
#> [1] 0.2200948
#> 
#> $perplexity
#> [1] 1.246195
#> 
#> $token_losses
#> [1] 0.3132617 0.1269280
#> 
#> $n_response_tokens
#> [1] 2
#> 
#> $vocab_size
#> [1] 2
#> 
#> $n
#> [1] 3
#> 
#> $method
#> [1] "Instruction-tuning CE over response tokens"
#>