Bot, or not? Comparing three methods for detecting social bots in five political discourses

• Published in: Big data & society Vol. 8; no. 2
• Main Authors: Martini, Franziska, Samula, Paul, Keller, Tobias R, Klinger, Ulrike
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.07.2021; Sage Publications Ltd; SAGE Publishing
• Subjects: Automation; Comparative analysis; Discourse; Interdisciplinary aspects; Interdisciplinary studies; Political systems; Politics; Social dynamics; Social media; Software agents; Triangulation; Twitter; bot detection; comparative research; Social bots; political discourse
• ISSN: 2053-9517; 2053-9517
• DOI: 10.1177/20539517211033566

Social bots – partially or fully automated accounts on social media platforms – have not only been widely discussed, but have also entered political, media and research agendas. However, bot detection is not an exact science. Quantitative estimates of bot prevalence vary considerably and comparative research is rare. We show that findings on the prevalence and activity of bots on Twitter depend strongly on the methods used to identify automated accounts. We search for bots in political discourses on Twitter, using three different bot detection methods: Botometer, Tweetbotornot and “heavy automation”. We drew a sample of 122,884 unique user Twitter accounts that had produced 263,821 tweets contributing to five political discourses in five Western democracies. While all three bot detection methods classified accounts as bots in all our cases, the comparison shows that the three approaches produce very different results. We discuss why neither manual validation nor triangulation resolves the basic problems, and conclude that social scientists studying the influence of social bots on (political) communication and discourse dynamics should be careful with easy-to-use methods, and consider interdisciplinary research.