Real-Time Deepfake Video Detection Using Eye Movement Analysis with a Hybrid Deep Learning Approach

• Published in: Electronics (Basel) Vol. 13; no. 15; p. 2947
• Main Authors: Javed, Muhammad, Zhang, Zhaohui, Dahri, Fida Hussain, Laghari, Asif Ali
• Format: Journal Article
• Language: English
• Published: Basel MDPI AG 01.08.2024
• Subjects: Accuracy; Artificial intelligence; Artificial neural networks; Deception; Deep learning; Deepfake; Eye movements; Feature extraction; Image enhancement; Image manipulation; Light; Lighting; Machine learning; Motion perception; Neural networks; Privacy; Probability; Real time; Retina; Robustness (mathematics); Social networks; Streaming media; Technology application; Technology assessment; Video data; China
• ISSN: 2079-9292; 2079-9292
• DOI: 10.3390/electronics13152947

Deepfake technology uses artificial intelligence to create realistic but false audio, images, and videos. Deepfake technology poses a significant threat to the authenticity of visual content, particularly in live-stream scenarios where the immediacy of detection is crucial. Existing Deepfake detection approaches have limitations and challenges, prompting the need for more robust and accurate solutions. This research proposes an innovative approach: combining eye movement analysis with a hybrid deep learning model to address the need for real-time Deepfake detection. The proposed hybrid deep learning model integrates two deep neural network architectures, MesoNet4 and ResNet101, to leverage their respective architectures’ strengths for effective Deepfake classification. MesoNet4 is a lightweight CNN model designed explicitly to detect subtle manipulations in facial images. At the same time, ResNet101 handles complex visual data and robust feature extraction. Combining the localized feature learning of MesoNet4 with the deeper, more comprehensive feature representations of ResNet101, our robust hybrid model achieves enhanced performance in distinguishing between manipulated and authentic videos, which cannot be conducted with the naked eye or traditional methods. The model is evaluated on diverse datasets, including FaceForensics++, CelebV1, and CelebV2, demonstrating compelling accuracy results, with the hybrid model attaining an accuracy of 0.9873 on FaceForensics++, 0.9689 on CelebV1, and 0.9790 on CelebV2, showcasing its robustness and potential for real-world deployment in content integrity verification and video forensics applications.