Cut-and-Paste: Subject-driven video editing with attention control

• Published in: Neural networks Vol. 181; p. 106818
• Main Authors: Zuo, Zhichao, Zhang, Zhao, Luo, Yan, Zhao, Yang, Zhang, Haijun, Yang, Yi, Wang, Meng
• Format: Journal Article
• Language: English
• Published: United States Elsevier Ltd 01.01.2025
• Subjects: Attention - physiology; Attention control with adjacent frames; Fine-grained semantic edits; Humans; Image Processing, Computer-Assisted - methods; Neural Networks, Computer; Semantics; Subject driven; Video editing; Video editing; Subject driven; Fine-grained semantic edits; Attention control with adjacent frames
• ISSN: 0893-6080; 1879-2782; 1879-2782
• DOI: 10.1016/j.neunet.2024.106818

This paper presents a novel framework termed Cut-and-Paste for real-word semantic video editing under the guidance of text prompt and additional reference image. While the text-driven video editing has demonstrated remarkable ability to generate highly diverse videos following given text prompts, the fine-grained semantic edits are hard to control by plain textual prompt only in terms of object details and edited region, and cumbersome long text descriptions are usually needed for the task. We therefore investigate subject-driven video editing for more precise control of both edited regions and background preservation, and fine-grained semantic generation. We achieve this goal by introducing an reference image as supplementary input to the text-driven video editing, which avoids racking your brain to come up with a cumbersome text prompt describing the detailed appearance of the object. To limit the editing area, we refer to a method of cross attention control in image editing and successfully extend it to video editing by fusing the attention map of adjacent frames, which strikes a balance between maintaining video background and spatio-temporal consistency. Compared with current methods, the whole process of our method is like “cut” the source object to be edited and then “paste” the target object provided by reference image. We demonstrate that our method performs favorably over prior arts for video editing under the guidance of text prompt and extra reference image, as measured by both quantitative and subjective evaluations.