Character-level RNN language model (Geron Ch 16, morie.fn hmchrn)

A vanilla RNN trained with FULL backpropagation through time on the single sequence text. The vocabulary is the sorted unique characters in codepoint order (method = "radix", matching Python's sorted(..., key=str); R's locale collation would NOT match). Weights come from one continuing LCG stream: Wxh (V x H) then Whh (H x H), each reshaped row-major.

Usage

morie_geron_char_rnn(
  text,
  hidden = 8,
  epochs = 50,
  lr = 0.1,
  seed = 0,
  generate = 0
)

Arguments

text: Character scalar of at least 2 characters.
hidden: Hidden width.
epochs: Training epochs.
lr: Learning rate.
seed: LCG seed.
generate: Number of characters to sample greedily.

Value

List with loss_history, final_loss, chance_loss, perplexity, vocab, vocab_size, weights, hidden, generated, estimate, n.

Details

A uniform model scores log V, so final_loss below that is structure the RNN has captured.

Examples

D <- data.frame(x = c(1, 2, 3, 4), y = c(2, 4, 5, 9))
morie_geron_char_rnn(D)
#> $loss_history
#>  [1] 2.197225 2.179463 2.162033 2.144885 2.127975 2.111258 2.094692 2.078238
#>  [9] 2.061854 2.045501 2.029139 2.012728 1.996228 1.979596 1.962791 1.945768
#> [17] 1.928484 1.910895 1.892957 1.874627 1.855867 1.836639 1.816917 1.796678
#> [25] 1.775913 1.754622 1.732822 1.710543 1.687830 1.664740 1.641345 1.617725
#> [33] 1.593966 1.570160 1.546394 1.522758 1.499332 1.476193 1.453407 1.431033
#> [41] 1.409117 1.387701 1.366813 1.346476 1.326704 1.307506 1.288882 1.270829
#> [49] 1.253341 1.236406
#> 
#> $final_loss
#> [1] 1.220012
#> 
#> $chance_loss
#> [1] 2.197225
#> 
#> $perplexity
#> [1] 3.387227
#> 
#> $vocab
#> [1] " " "(" ")" "," "1" "2" "3" "4" "c"
#> 
#> $vocab_size
#> [1] 9
#> 
#> $weights
#> $weights$Wxh
#>              [,1]        [,2]        [,3]        [,4]        [,5]        [,6]
#>  [1,] -0.36691297 -0.11313192  0.37973210  0.27461029 -0.15466253  0.27652923
#>  [2,] -0.08127538  0.21362994  0.27945516  0.35231952  0.06934507  0.59900232
#>  [3,]  0.23986927 -0.21745876 -0.13392389  0.16706021  0.02870011  0.04815724
#>  [4,] -0.60810151 -0.25981238  0.69010415  0.05427856  0.14571245 -0.57249574
#>  [5,] -0.13313673 -0.28551997  0.53939438  0.09500078  0.04001400 -0.42293940
#>  [6,] -0.31977803 -0.52354549  0.09324055  0.35180651 -0.53806954 -0.40398113
#>  [7,] -0.21721340 -0.07817314 -0.43637920 -0.40147037  0.38236599 -0.06606737
#>  [8,] -0.06954927  0.01766720 -0.55177060 -0.02863297 -0.40039229  0.43180064
#>  [9,] -0.06090708 -0.22100539 -0.20348475  0.34253573 -0.24794621 -0.48802754
#>             [,7]         [,8]
#>  [1,] -0.1427014  0.256213139
#>  [2,] -0.6369620  0.079769686
#>  [3,] -0.1151093 -0.291941458
#>  [4,] -0.1587469 -0.646941258
#>  [5,]  0.4365705 -0.005783432
#>  [6,] -0.4939082 -0.083028487
#>  [7,] -0.5717209  0.125695237
#>  [8,] -0.4095419 -0.053565324
#>  [9,]  0.4975184  0.411431574
#> 
#> $weights$Whh
#>             [,1]         [,2]        [,3]        [,4]        [,5]        [,6]
#> [1,] -0.18506384 -0.292301262 -0.02373480  0.48388071  0.60346642 -0.36168806
#> [2,]  0.52595851 -0.148843968 -0.19847760 -0.10013238 -0.77940499  0.17896951
#> [3,] -0.40556320  0.165678833 -0.18432059  0.49741758 -0.27266897  0.21561749
#> [4,]  0.06542186 -0.430965713  0.06440655  0.03393508 -0.16606023  0.36251935
#> [5,]  0.35486723  0.009521258  0.57366815  0.63817968 -0.58489754  0.55008722
#> [6,]  0.21074898 -0.695975500  0.02371661 -0.05165702 -0.20076585  0.03983041
#> [7,]  0.22536892  0.494356488 -0.50359057  0.41487232 -0.22637775  0.35512258
#> [8,] -0.18400428 -0.423981092 -0.44497246 -0.34598570 -0.03828117  0.08227787
#>            [,7]        [,8]
#> [1,]  0.5353298  0.48744135
#> [2,] -0.4540016  0.14318118
#> [3,]  0.3884200 -0.47383174
#> [4,]  0.0228691 -0.47205043
#> [5,]  0.5947145  0.57614975
#> [6,] -0.4103939 -0.53562769
#> [7,]  0.1860715  0.04806218
#> [8,]  0.4602847 -0.37767702
#> 
#> $weights$bh
#> [1] -0.165172244  0.062941148  0.040657151 -0.005039746 -0.045537522
#> [6]  0.036612443 -0.015966804 -0.055353863
#> 
#> $weights$Why
#>             [,1]        [,2]        [,3]        [,4]        [,5]        [,6]
#> [1,] -0.41000579  0.11686716  0.09638554 -0.05971734  0.07523658  0.14786230
#> [2,]  0.01656726 -0.09037134  0.04388884 -0.55628247  0.09882140  0.20234782
#> [3,]  0.48825049 -0.18729015 -0.35287324 -0.16905555 -0.12125644  0.14459971
#> [4,] -0.18038040  0.11908633 -0.12810434 -0.17197131 -0.01201803  0.16488389
#> [5,]  0.49204185 -0.02781935 -0.17862369 -0.23979574  0.23707452 -0.16146592
#> [6,] -0.62792334 -0.14957362  0.24415863 -0.11312343  0.20157327  0.24813748
#> [7,] -0.22241646  0.26404770 -0.09998667 -0.28765359 -0.01973343 -0.01456895
#> [8,] -0.30597431  0.23264511 -0.17141112 -0.39016759 -0.00612091  0.14597202
#>              [,7]        [,8]         [,9]
#> [1,] -0.002239944 -0.10950213  0.145113620
#> [2,]  0.200571677  0.08117258  0.003284243
#> [3,]  0.180319654  0.10201013 -0.084704598
#> [4,]  0.140497442  0.07544039 -0.007433970
#> [5,] -0.159659452 -0.04160171  0.079849491
#> [6,]  0.126698342  0.05408309  0.015969577
#> [7,]  0.065917958  0.20010520  0.114288248
#> [8,]  0.257052056  0.16196140  0.076043358
#> 
#> $weights$by
#> [1]  0.35578885 -0.03321664 -0.07947311  0.44344025 -0.04959906 -0.08319057
#> [7] -0.09060930 -0.07274111 -0.39039931
#> 
#> 
#> $hidden
#> [1] 8
#> 
#> $generated
#> [1] ""
#> 
#> $estimate
#> [1] 1.220012
#> 
#> $n
#> [1] 12
#> 
#> $method
#> [1] "vanilla char-RNN trained with full backpropagation through time"
#>