Safe Pretraining of Deep Language Models in a Synthetic Pseudo-Language
Absract This paper compares the pretraining of a transformer on natural language texts and on sentences of a synthetic pseudo-language. The artificial texts are automatically generated according to the rules written in a context-free grammar. The results of fine-tuning to complete tasks of the Russi...
Saved in:
Published in | Doklady. Mathematics Vol. 108; no. Suppl 2; pp. S494 - S502 |
---|---|
Main Authors | , |
Format | Journal Article |
Language | English |
Published |
Moscow
Pleiades Publishing
01.12.2023
Springer Nature B.V |
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Absract
This paper compares the pretraining of a transformer on natural language texts and on sentences of a synthetic pseudo-language. The artificial texts are automatically generated according to the rules written in a context-free grammar. The results of fine-tuning to complete tasks of the RussianSuperGLUE project statistically reliably showed that the models had the same scores. That is, the use of artificial texts facilitates the AI safety, because it can completely control the composition of the dataset. In addition, at the pretraining stage of a RoBERTa-like model, it is enough to learn recognizing only the syntactic and morphological patterns of the language, which can be successfully created in a fairly simple way, such as a context-free grammar. |
---|---|
ISSN: | 1064-5624 1531-8362 |
DOI: | 10.1134/S1064562423701636 |