TD(0) value update (Geron Ch 18, morie.fn hmtd)

Transitions are applied SEQUENTIALLY, so a repeated state sees its own earlier update – the parity test replays a trajectory step-for-step to pin that down. States are 0-based.

Usage

morie_geron_td_learning(
  V,
  s,
  r,
  s_next,
  alpha = 0.1,
  gamma = 0.9,
  terminal = NULL
)

Arguments

V

Value table.

s, r, s_next

Equal-length transitions.

alpha

In (0, 1].

gamma

In [0, 1].

terminal

Optional logical vector; TRUE drops the bootstrap.

Value

List with V, td_error, target, updates, alpha, gamma, estimate, n.

Examples

morie_geron_td_learning(c(0, 0, 0), c(0, 1, 0, 2), c(1, 0, 2,
    -1), c(1, 2, 1, 2), alpha = 0.5, gamma = 0.9, terminal = c(FALSE,
    FALSE, FALSE, TRUE))
#> $V
#> [1]  1.25  0.00 -0.50
#> 
#> $td_error
#> [1]  1.0  0.0  1.5 -1.0
#> 
#> $target
#> [1]  1  0  2 -1
#> 
#> $updates
#> [1] 4
#> 
#> $alpha
#> [1] 0.5
#> 
#> $gamma
#> [1] 0.9
#> 
#> $estimate
#> [1] 0.875
#> 
#> $n
#> [1] 4
#> 
#> $method
#> [1] "TD(0) value update applied sequentially"
#>