Language autodetection from non-character sub-token signals

• Main Authors: Radtke, Roland, Glass, Andrew Stuart, Magnus, Margaret Hope
• Format: Patent
• Language: English
• Published: 14.06.2022
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

In non-limiting examples of the present disclosure, systems, methods and devices for determining a language of a text string are presented. A language detection model may be maintained. The language detection model may comprise identities and weights for initial and final consonants, identities and weights for prefixes and suffixes, and identities and weights for vowel sequences, where each identity is derived from a training corpus. The weights may correspond to a frequency of a text unit in the corpus. A text string may be received and a match score between the text string and the language of the language detection model may be determined. The match score may be based on initial and final consonant scores, prefix and suffix scores, and/or vowel sequence scores for each word in the text string. If the match score meets a threshold value a follow-up action associated with the language may be performed.