Detecting images generated by diffusers

• Published in: PeerJ. Computer science Vol. 10; p. e2127
• Main Authors: Coccomini, Davide Alessandro, Esuli, Andrea, Falchi, Fabrizio, Gennaro, Claudio, Amato, Giuseppe
• Format: Journal Article
• Language: English
• Published: United States PeerJ. Ltd 10.07.2024; PeerJ Inc
• Subjects: Artificial Intelligence; CLIP; Computational linguistics; Computer Vision; Data Mining and Machine Learning; Deep learning; Deepfake detection; Language processing; Multimodal machine learning; Natural language interfaces; Neural Networks; Synthetic image detection; Deep learning; Computer vision; Deepfake detection; Transformers; Synthetic image detection; Convolutional neural networks; Multimodal machine learning; CLIP
• ISSN: 2376-5992; 2376-5992
• DOI: 10.7717/peerj-cs.2127

In recent years, the field of artificial intelligence has witnessed a remarkable surge in the generation of synthetic images, driven by advancements in deep learning techniques. These synthetic images, often created through complex algorithms, closely mimic real photographs, blurring the lines between reality and artificiality. This proliferation of synthetic visuals presents a pressing challenge: how to accurately and reliably distinguish between genuine and generated images. This article, in particular, explores the task of detecting images generated by text-to-image diffusion models, highlighting the challenges and peculiarities of this field. To evaluate this, we consider images generated from captions in the MSCOCO and Wikimedia datasets using two state-of-the-art models: Stable Diffusion and GLIDE. Our experiments show that it is possible to detect the generated images using simple multi-layer perceptrons (MLPs), starting from features extracted by CLIP or RoBERTa, or using traditional convolutional neural networks (CNNs). These latter models achieve remarkable performances in particular when pretrained on large datasets. We also observe that models trained on images generated by Stable Diffusion can occasionally detect images generated by GLIDE, but only on the MSCOCO dataset. However, the reverse is not true. Lastly, we find that incorporating the associated textual information with the images in some cases can lead to a better generalization capability, especially if textual features are closely related to visual ones. We also discovered that the type of subject depicted in the image can significantly impact performance. This work provides insights into the feasibility of detecting generated images and has implications for security and privacy concerns in real-world applications. The code to reproduce our results is available at: https://github.com/davide-coccomini/Detecting-Images-Generated-by-Diffusers .