DistilBERT triple-loss distillation (hmdbrt) — morie_geron

Distillation KL (T^2-scaled) + MLM cross-entropy + optional cosine hidden-state term; architectures resolved via morie_geron_encoder_only (hmencox) for the real 12- vs 6-layer parameter reduction.

Usage

morie_geron_distilbert(
  teacher,
  student,
  X,
  temperature = 2,
  alpha_ce = 0.5,
  alpha_mlm = 0.5,
  alpha_cos = 0,
  mlm_labels = NULL
)

Arguments

teacher, student: Logits matrix (B, C), or list(logits, hidden).
X: Inputs (only its length is used for the token count).
temperature: Distillation temperature (>=1).
alpha_ce, alpha_mlm, alpha_cos: Non-negative loss weights.
mlm_labels: 0-based labels, required if alpha_mlm > 0.

Value

List with loss, loss_ce, loss_mlm, loss_cos, teacher_params, student_params, param_reduction, agreement.

Examples

morie_geron_distilbert(matrix(c(2, 0), 1, 2), matrix(c(2, 0),
    1, 2), c(1, 2, 3), alpha_mlm = 0, alpha_ce = 1)
#> $loss
#> [1] 0
#> 
#> $loss_ce
#> [1] 0
#> 
#> $loss_mlm
#> [1] 0
#> 
#> $loss_cos
#> [1] 0
#> 
#> $teacher_params
#> [1] 108891648
#> 
#> $student_params
#> [1] 66364416
#> 
#> $param_reduction
#> [1] 0.3905463
#> 
#> $agreement
#> [1] 1
#> 
#> $temperature
#> [1] 2
#> 
#> $weights
#> $weights$ce
#> [1] 1
#> 
#> $weights$mlm
#> [1] 0
#> 
#> $weights$cos
#> [1] 0
#> 
#> 
#> $estimate
#> [1] 0
#> 
#> $n
#> [1] 1
#> 
#> $method
#> [1] "DistilBERT triple loss with T^2-scaled KL; architectures resolved through hmencox"
#>