A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models
The robustness of Vision-Language Models (VLMs) such as CLIP is critical for their deployment in safety-critical applications like autonomous driving, healthcare diagnostics, and security systems, where accurate interpretation of visual and textual data is essential. However, these models are highly...
Saved in:
Main Authors | , , , , |
---|---|
Format | Journal Article |
Language | English |
Published |
18.10.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | The robustness of Vision-Language Models (VLMs) such as CLIP is critical for
their deployment in safety-critical applications like autonomous driving,
healthcare diagnostics, and security systems, where accurate interpretation of
visual and textual data is essential. However, these models are highly
susceptible to adversarial attacks, which can severely compromise their
performance and reliability in real-world scenarios. Previous methods have
primarily focused on improving robustness through adversarial training and
generating adversarial examples using models like FGSM, AutoAttack, and
DeepFool. However, these approaches often rely on strong assumptions, such as
fixed perturbation norms or predefined attack patterns, and involve high
computational complexity, making them challenging to implement in practical
settings. In this paper, we propose a novel adversarial training framework that
integrates multiple attack strategies and advanced machine learning techniques
to significantly enhance the robustness of VLMs against a broad range of
adversarial attacks. Experiments conducted on real-world datasets, including
CIFAR-10 and CIFAR-100, demonstrate that the proposed method significantly
enhances model robustness. The fine-tuned CLIP model achieved an accuracy of
43.5% on adversarially perturbed images, compared to only 4% for the baseline
model. The neural network model achieved a high accuracy of 98% in these
challenging classification tasks, while the XGBoost model reached a success
rate of 85.26% in prediction tasks. |
---|---|
DOI: | 10.48550/arxiv.2410.14911 |