GPT-2 decoder-only language model

The decoder-only transformer forward pass at the four released GPT-2 sizes (BPE vocabulary 50257, context 1024); the architecture is the masked self-attention decoder of GPT scaled up. There is exactly one implementation: this function delegates to morie_geron_gpt2.

Usage

Gpt2(X, n_layers = NULL, n_heads = NULL, size = "small", ...)

Arguments

X

Input token embedding matrix.

n_layers

Optional depth override.

n_heads

Optional head-count override.

size

One of small, medium, large, xl. Default small.

...

Further configuration passed through.

Value

List as returned by morie_geron_gpt2.

References

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019), Language Models are Unsupervised Multitask Learners, OpenAI technical report, Section 2.3 and Table 2. Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. (2018), Improving Language Understanding by Generative Pre-Training, OpenAI.

Examples

Gpt2(matrix(0.1, 2, 4), n_layers = 1, n_heads = 1)$estimate
#> [1] 46473216