A Novel Approach: Tokenization Framework based on Sentence Structure in Indonesian Language

• Published in: International journal of advanced computer science & applications Vol. 14; no. 2
• Main Authors: Petrus, Johannes, -, Ermatita, -, Sukemi, -, Erwin
• Format: Journal Article
• Language: English
• Published: West Yorkshire Science and Information (SAI) Organization Limited 2023
• Subjects: Computer science; Datasets; Sentences; Training; Words (language)
• ISSN: 2158-107X; 2156-5570
• DOI: 10.14569/IJACSA.2023.0140264

This study proposes a new approach in the sentence tokenization process. Sentence tokenization, which is known so far, is the process of breaking sentences based on spaces as separators. Space-based sentence tokenization only generates single word tokens. In sentences consisting of five words, token...