Exploring Edit Distance for Normalising Out-of-Vocabulary Malay Words on Social Media

Users interact using short-formed words and abbreviations and this results in a message full of noisy words that are not recognized by the system's knowledge. The aim of this research is to overcome the limitations that still bar the progression of normalizing Malay noisy words from social medi...

Full description

Saved in:
Bibliographic Details
Published inMATEC Web of Conferences Vol. 255; p. 3001
Main Authors Raja, Roza Athirah, Lay-Ki, Soon, Su-Cheng, Haw
Format Journal Article Conference Proceeding
LanguageEnglish
Published Les Ulis EDP Sciences 2019
Subjects
Abbreviations
Digital media
Normalizing
Social networks
Words (language)
Online AccessGet full text

Cover

More Information
Summary:Users interact using short-formed words and abbreviations and this results in a message full of noisy words that are not recognized by the system's knowledge. The aim of this research is to overcome the limitations that still bar the progression of normalizing Malay noisy words from social media platforms. The testing data gathered is 25,000; 15,000 Tweets from Twitter and 10,000 comments from Facebook respectively. Pre-processing steps were carried out to clean the entire dataset which consists of unique 179,786 words. 36,587 out-of-vocabulary (OOV) Malay terms were then extracted and checked against an in- vocabulary (IV) Malay corpus using the Levenshtein edit distance formula and character manipulation rules. The resultant output is 3,964 unique IV Malay words. Based on the results, the usage of edit distance and rules can be further improved to elevate the normalisation of the ever changing colloquial terms of the Malay language.
ISSN:2261-236X
2274-7214
2261-236X
DOI:10.1051/matecconf/201925503001