REINFORCE Monte Carlo policy gradient (Geron Ch 18, morie.fn grrein)

theta += alpha sum_t (G_t - b) grad log pi_t; log_probs must be score VECTORS, one row per step.

Usage

morie_geron_reinforce_policy_gradient(
  theta,
  log_probs,
  returns_G,
  alpha,
  baseline = NULL
)

Arguments

theta

Parameters.

log_probs

(T, p) score matrix.

returns_G

Length-T returns.

alpha

Positive step size.

baseline

NULL, "mean", or a finite number.

Value

List with theta_new, gradient, advantages, baseline, step_norm.

Examples

set.seed(2)
S <- matrix(rnorm(15), 5, 3)
G <- c(1, 0.9, 0.8, 0.7, 0.6)
morie_geron_reinforce_policy_gradient(c(0, 0, 0), S, G, alpha = 0.01,
                                      baseline = "mean")
#> $theta_new
#> [1] -0.0003181007 -0.0007341046 -0.0007077347
#> 
#> $gradient
#> [1] -0.03181007 -0.07341046 -0.07077347
#> 
#> $advantages
#> [1]  0.2  0.1  0.0 -0.1 -0.2
#> 
#> $baseline
#> [1] 0.8
#> 
#> $step_norm
#> [1] 0.001068169
#> 
#> $estimate
#> [1] -0.0003181007 -0.0007341046 -0.0007077347
#> 
#> $n
#> [1] 5
#>