TD(0) value update (Geron Ch 18, morie.fn grtd0)

V(s) += alpha [r + gamma V(s') - V(s)]; done drops the bootstrap. States are 0-based.

Usage

morie_geron_td_zero_update(
  V,
  state,
  next_state,
  reward,
  alpha,
  gamma,
  done = FALSE
)

Arguments

V

State value vector.

state, next_state

0-based indices.

reward

Finite reward.

alpha

Step size in (0, 1].

gamma

Discount in [0, 1].

done

Terminal flag.

Value

List with V, old_value, new_value, target, td_error.

Examples

morie_geron_td_zero_update(c(0, 1, -1), 0, 2, 0.5, 0.3, 0.9)
#> $V
#> [1] -0.12  1.00 -1.00
#> 
#> $old_value
#> [1] 0
#> 
#> $new_value
#> [1] -0.12
#> 
#> $target
#> [1] -0.4
#> 
#> $td_error
#> [1] -0.4
#> 
#> $estimate
#> [1] -0.12
#> 
#> $n
#> [1] 3
#>