Tuning Traditional Language Processing Approaches for Pashto Text Classification

• Published in: arXiv.org
• Main Authors: Jawid Ahmad Baktash, Dawodi, Mursal, Mohammad Zarif Joya, Hassanzada, Nematullah
• Format: Paper Journal Article
• Language: English
• Published: Ithaca Cornell University Library, arXiv.org 04.05.2023
• Subjects: Algorithms; Classification; Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Learning; Decision trees; Documents; Feature extraction; Machine learning; Multilayer perceptrons; Natural language processing; Neural networks; Statistical analysis; Support vector machines; Text categorization
• ISSN: 2331-8422
• DOI: 10.48550/arxiv.2305.03737

Today text classification becomes critical task for concerned individuals for numerous purposes. Hence, several researches have been conducted to develop automatic text classification for national and international languages. However, the need for an automatic text categorization system for local languages is felt. The main aim of this study is to establish a Pashto automatic text classification system. In order to pursue this work, we built a Pashto corpus which is a collection of Pashto documents due to the unavailability of public datasets of Pashto text documents. Besides, this study compares several models containing both statistical and neural network machine learning techniques including Multilayer Perceptron (MLP), Support Vector Machine (SVM), K Nearest Neighbor (KNN), decision tree, gaussian na\"ive Bayes, multinomial na\"ive Bayes, random forest, and logistic regression to discover the most effective approach. Moreover, this investigation evaluates two different feature extraction methods including unigram, and Time Frequency Inverse Document Frequency (IFIDF). Subsequently, this research obtained average testing accuracy rate 94% using MLP classification algorithm and TFIDF feature extraction method in this context.