Language model automatic training method

• Main Authors: CHU JIANXIA, ZHOU JIAN, ZHANG ENQIAO
• Format: Patent
• Language: Chinese; English
• Published: 18.08.2023
• Subjects: ACOUSTICS; CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; MUSICAL INSTRUMENTS; PHYSICS; SPEECH ANALYSIS OR SYNTHESIS; SPEECH OR AUDIO CODING OR DECODING; SPEECH OR VOICE PROCESSING; SPEECH RECOGNITION

The invention provides a language model automatic training method, and relates to the technical field of natural language processing. The language model automatic training method comprises the following steps: S1, acquiring language model training data comprising a large number of language sample texts; s2, performing word segmentation processing on the sample text by using a language model without category labels to obtain word segmentation data without category labels corresponding to each segmented word, the word segmentation labels including position information of each character in the corresponding segmented word in the corresponding segmented word; and S3, performing related word class replacement on each piece of word segmentation data without class labels to obtain first word segmentation data with class labels. According to the method and the device, the language model is trained by taking the word segmentation labels of the sample text as the training data, so that the data used for training the la