Part of speech and gramset tagging algorithms for unknown words based on morphological dictionaries of the Veps and Karelian languages

• Main Authors: Krizhanovsky, Andrew, Krizhanovsky, Natalia, Novak, Irina
• Format: Journal Article
• Language: English
• Published: 22.03.2021
• Subjects: Computer Science - Computation and Language; Computer Science - Information Retrieval
• DOI: 10.48550/arxiv.2103.11859

This research devoted to the low-resource Veps and Karelian languages. Algorithms for assigning part of speech tags to words and grammatical properties to words are presented in the article. These algorithms use our morphological dictionaries, where the lemma, part of speech and a set of grammatical features (gramset) are known for each word form. The algorithms are based on the analogy hypothesis that words with the same suffixes are likely to have the same inflectional models, the same part of speech and gramset. The accuracy of these algorithms were evaluated and compared. 313 thousand Vepsian and 66 thousand Karelian words were used to verify the accuracy of these algorithms. The special functions were designed to assess the quality of results of the developed algorithms. 92.4% of Vepsian words and 86.8% of Karelian words were assigned a correct part of speech by the developed algorithm. 95.3% of Vepsian words and 90.7% of Karelian words were assigned a correct gramset by our algorithm. Morphological and semantic tagging of texts, which are closely related and inseparable in our corpus processes, are described in the paper.