PPO clipped surrogate objective (Geron Ch 19, morie.fn grppo)

L = mean min(rA, clip(r, 1-e, 1+e) A). clipped_fraction uses an isTRUE(all.equal)-style tolerance matching numpy's isclose defaults (rtol 1e-5, atol 1e-8).

Usage

morie_geron_ppo_clipped_objective(ratios, advantages, eps = 0.2)

Arguments

ratios

Positive probability ratios.

advantages

Advantage estimates.

eps

Clip half-width in (0, 1).

Value

List with objective, per_step, unclipped, clipped, clipped_fraction.

Examples

V <- c(1, 2, 3, 4, 5, 6, 7, 8)
morie_geron_ppo_clipped_objective(V, V)
#> $objective
#> [1] 5.375
#> 
#> $per_step
#> [1] 1.0 2.4 3.6 4.8 6.0 7.2 8.4 9.6
#> 
#> $unclipped
#> [1]  1  4  9 16 25 36 49 64
#> 
#> $clipped
#> [1] 1.0 2.4 3.6 4.8 6.0 7.2 8.4 9.6
#> 
#> $clipped_fraction
#> [1] 0.875
#> 
#> $estimate
#> [1] 5.375
#> 
#> $n
#> [1] 8
#>