Two transformer streams, one per modality, exchanging information
through co-attentional layers: image queries attend over text keys
and values while text queries attend over image keys and values.
There is exactly one implementation: this function delegates to
morie_geron_vilbert.
Usage
Vilbrt(image, text, d_model = 8, seed = 0)
Arguments
- image
Image-region feature matrix.
- text
Text-token feature matrix.
- d_model
Common projection width. Default 8.
- seed
RNG seed for the deterministic projection draw. Default 0.
Value
List as returned by morie_geron_vilbert.
References
Lu, J., Batra, D., Parikh, D. and Lee, S. (2019), ViLBERT: Pretraining
Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks,
NeurIPS 32, arXiv:1908.02265, Section 3.1 and Figure 2.
Examples
Vilbrt(matrix(0.1, 2, 3), matrix(0.2, 2, 3))$estimate
#> [1] 0.5
