Urdu Word Segmentation using Machine Learning Approaches

• Published in: International journal of advanced computer science & applications Vol. 9; no. 6
• Main Authors: Khan, Sadiq Nawaz, Khan, Khairullah, Khan, Wahab, Khan, Asfandyar, Subhan, Fazali, Ullah, Aman, Ullah, Burhan
• Format: Journal Article
• Language: English
• Published: West Yorkshire Science and Information (SAI) Organization Limited 2018
• Subjects: Conditional random fields; Data mining; Effectiveness; Languages; Machine learning; Segmentation; Sentiment analysis
• DOI: 10.14569/IJACSA.2018.090628
• ISSN: 2158-107X

Word Segmentation is considered a basic NLP task and in diverse NLP areas, it plays a significant role. The main areas which can be benefited from Word segmentation are IR, POS, NER, sentiment analysis, etc. Urdu Word Segmentation is a challenging task. There can be a number of reasons but Space Insertion Problem and Space Omission Problems are the major ones. Compared to Urdu, the tools and resources developed for word segmentation of English and English like other western languages have record-setting performance. Some languages provide a clear indication for words just like English which having space or capitalization of the first character in a word. But there are many languages which do not have proper delimitation in between words e.g. Thai, Lao, Urdu, etc. The objective of this research work is to present a machine learning based approach for Urdu word segmentation. We adopted the use of conditional random fields (CRF) to achieve the subject task. Some other challenges faced in Urdu text are compound words and reduplicated words. In this paper, we tried to overcome such challenges in Urdu text by machine learning methodology.