Owner name entity recognition in websites based on multiscale features and multimodal co-attention

• Published in: Expert systems with applications Vol. 224; p. 120014
• Main Authors: Ren, Yimo, Li, Hong, Liu, Peipei, Liu, Jie, Zhu, Hongsong, Sun, Limin
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 15.08.2023
• Subjects: Multimodal co-attention; Multimodal NER; Multiscale features; Network security; Owner NER; Multimodal co-attention; Network security; Owner NER; Multimodal NER; Multiscale features
• ISSN: 0957-4174; 1873-6793
• DOI: 10.1016/j.eswa.2023.120014

Identifying the owners of online devices on the Internet can enable numerous network security applications. For example, fast and accurate Owner Name Entity Recognition (ONER) of websites is critical to find influenced owners in light of new security threats. In this situation, as a specific task of Multimodal Named Entity Recognition (MNER), ONER is essential and helpful for network security. Currently, most of the existing MNER models only use texts and images, so they cannot effectively utilize the multimodal data of devices to achieve ONER accurately and fast. In order to improve performance and training speed simultaneously, the paper proposes a framework MFMCA: Multiscale Features and Multimodal Co-Attention. MFMCA is based on a two-step gated co-attention with fewer transformer blocks and simultaneously uses texts, images, and domains. Also, MFMCA extracts multiscale image features to get a fine-grained hint for ONER. Moreover, due to the lack of MNER datasets, the paper manually labels a multimodal dataset containing texts, images, and domains for MNER research. The experiments show that MFMCA achieves 0.8211 F1 scores on the recognition of owner entities, which is competitive compared with 0.8288, the best performance of existing state-of-the-art MNER models. However, MFMCA saves about 34% training time on the proposed dataset. •This paper raises the problem of Owner Name Entity Recognition of websites.•This paper constructs a manually labelled multimodal dataset, containing about 15,000 samples.•This paper designs a two-step gated co-attention to improve the performance of ONER.•The results show the competitive performance of our method with less cost time.•This paper builds multiscale image features to realize feature alignment well.