A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection

• Published in: Journal of systems and information technology Vol. 20; no. 3; pp. 358 - 374
• Main Authors: Teh, Phoey Lee, Ooi, Pei Boon, Chan, Nee Nee, Chuah, Yee Kang
• Format: Journal Article
• Language: English
• Published: Bingley Emerald Publishing Limited 14.11.2018; Emerald Group Publishing Limited
• Subjects: Coders; Commercialization; Comparative studies; Digital media; Empirical analysis; Human performance; Machine learning; Researchers; Sensitivity enhancement; Sentiment analysis; Social networks; Studies; Systems analysis; Texts; Verbal irony; Linguistic; Detection; Sarcasm; Social media
• ISSN: 1328-7265; 1758-8847
• DOI: 10.1108/JSIT-12-2017-0120

Purpose Sarcasm is often used in everyday speech and writing and is prevalent in online contexts. The purpose of this paper is to investigate the analogy between sarcasm comments from sentiment tools and the human coder. Design/methodology/approach Using the Verbal Irony Procedure, eight human coders were engaged to analyse comments collected from an online commercial page, and a dissimilarity analysis was conducted with sentiment tools. Three constants were tested, namely, polarity from sentiment tools, polarity rating by human coders; and sarcasm-level ratings by human coders. Findings Results found an inconsistent ratio between these three constants. Sentiment tools used did not have the capability or reliability to detect the subtle, contextualized meanings of sarcasm statements that human coders could detect. Further research is required to refine the sentiment tools to enhance their sensitivity and capability. Practical implications With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – for example, to incorporate sophisticated human sarcasm texts in their analytical systems. Sarcasm exists frequently in media, politics and human forms of communications in society. Therefore, more highly sophisticated sentiment tools with the abilities to detect human sarcasm would be vital in research and industry. Social implications The findings suggest that presently, of the sentiment tools investigated, most are still unable to pick up subtle contexts within the text which can reverse or change the message that the writer intends to send to his/her receiver. Hence, the use of the relevant hashtags (e.g. #sarcasm; #irony) are of fundamental importance in detection tools. This would aid the evaluation of product reviews online for commercial usage. Originality/value The value of this study lies in its original, empirical findings on the inconsistencies between sentiment tools and human coders in sarcasm detection. The current study proves these inconsistencies are detected between human and sentiment tools in social media texts and points to the inadequacies of current sentiment tools. With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – to incorporate sophisticated human sarcasm texts in their analytical systems. The system can then be used as a reference for psychologists, media analysts, researchers and speech writers to detect cues in the inconsistencies in behaviour and language.