MarianCG: a code generation transformer model inspired by machine translation

• Published in: Journal of engineering and applied science (Online) Vol. 69; no. 1; pp. 1 - 23
• Main Authors: Soliman, Ahmed S., Hadhoud, Mayada M., Shaheen, Samir I.
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.12.2022; Springer Nature B.V; SpringerOpen
• Subjects: Accuracy; Code generation; CoNaLa; Datasets; Deep learning; Embedding; Engineering; Machine translation; Marian NMT; MarianCG; MarianMT; Natural language; Natural language (computers); Natural language programming; Programming languages; Researchers; Semantics; Software; Transformers; Code generation; Marian NMT; CoNaLa; Natural language programming; Neural machine translation; MarianCG; MarianMT
• ISSN: 1110-1903; 2536-9512
• DOI: 10.1186/s44147-022-00159-4

The idea that computers can build their own programs is extremely significant, and many researchers are working on this challenge. Code generation is described as the process of generating executable code that can be run directly on the computer and fulfills the natural language requirements. It is an intriguing topic that might assist developers to learn a new software technology or programming language, or it could be a simple technique to help in coding through the description of the natural language code developer. In this paper, we present MarianCG, a code generation Transformer model used to tackle the code generation challenge of generating python code from natural language descriptions. Marian neural machine translation (NMT), which is the core model of the Microsoft Translator, is the basis for our NL-to-Code translation engine and is the heart of the teaching model. MarianMT is the teacher language model in our study, and it is one of the most successful machine translation transformers. In our approach, we use a sinusoidal positional embedding technique to represent the position of each token in the text, as well as no layer normalization embedding. Our code generation approach, MarianCG, is based on fine-tuning a machine translation pre-trained language model. This allows us to demonstrate that the pre-trained translation model can also operate and work as a code generation model. The proposed model outperforms recent state-of-the-art models in the problem of code generation when trained on the CoNaLa and DJANGO datasets. MarianCG model scores a BLEU score of 34.43 and an exact match accuracy of 10.2% on the CoNaLa dataset. Also, this model records a BLEU score of 90.41 and an exact match accuracy of 81.83% on the DJANGO dataset. The implementation of MarianCG model and relevant resources are available at  https://www.github.com/AhmedSSoliman/MarianCG-NL-to-Code .