Variance of a training-loss stream, keeping the correlation — morie_loss_stream

Losses recorded along a training run are NOT independent draws: consecutive steps share network weights and, in self-play, share the replay buffer the batches come from. The usual \(s/\sqrt{n}\) treats them as independent and is therefore too small, often by a large factor, which makes ordinary drift look like an improvement. Dividing by the effective sample size instead gives an inflation of \(\sqrt{\tau_{int}}\), which is 1 on an independent stream.

Usage

morie_loss_stream_variance(losses, alpha = 0.05)

Arguments

losses: Total loss at each recorded step, IN ORDER.
alpha: Two-sided level.

Value

A list with estimate, se, se_naive, se_inflation, ess, tau_int, ci_lower, ci_upper, warnings.

References

Silver D et al (2018) Science 362:1140-1144. Geyer CJ (1992) Statistical Science 7(4):473-483.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_loss_stream_variance(V)
#> $estimate
#> [1] 4.5
#> 
#> $variance
#> [1] 6
#> 
#> $se
#> [1] 1.448521
#> 
#> $se_naive
#> [1] 0.8660254
#> 
#> $se_inflation
#> [1] 1.672608
#> 
#> $ess
#> [1] 2.859574
#> 
#> $tau_int
#> [1] 2.797619
#> 
#> $ci_lower
#> [1] 1.66095
#> 
#> $ci_upper
#> [1] 7.33905
#> 
#> $ci_naive_lower
#> [1] 2.802621
#> 
#> $ci_naive_upper
#> [1] 6.197379
#> 
#> $n
#> [1] 8
#> 
#> $warnings
#> [1] "The loss stream is strongly autocorrelated (integrated time 2.8). Treating the 8 recorded steps as independent would understate the standard error by a factor of 1.67."
#> [2] "The effective sample size is only 2.9. The mean is not well determined by this run however many steps were logged."                                                     
#> 
#> $method
#> [1] "Autocorrelation-aware variance of a training loss stream"
#>