Reference-based Controllable Scene Stylization with Gaussian Splatting
Referenced-based scene stylization that edits the appearance based on a content-aligned reference image is an emerging research area. Starting with a pretrained neural radiance field (NeRF), existing methods typically learn a novel appearance that matches the given style. Despite their effectiveness...
Saved in:
Main Authors | , , |
---|---|
Format | Journal Article |
Language | English |
Published |
09.07.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Summary: | Referenced-based scene stylization that edits the appearance based on a
content-aligned reference image is an emerging research area. Starting with a
pretrained neural radiance field (NeRF), existing methods typically learn a
novel appearance that matches the given style. Despite their effectiveness,
they inherently suffer from time-consuming volume rendering, and thus are
impractical for many real-time applications. In this work, we propose ReGS,
which adapts 3D Gaussian Splatting (3DGS) for reference-based stylization to
enable real-time stylized view synthesis. Editing the appearance of a
pretrained 3DGS is challenging as it uses discrete Gaussians as 3D
representation, which tightly bind appearance with geometry. Simply optimizing
the appearance as prior methods do is often insufficient for modeling
continuous textures in the given reference image. To address this challenge, we
propose a novel texture-guided control mechanism that adaptively adjusts local
responsible Gaussians to a new geometric arrangement, serving for desired
texture details. The proposed process is guided by texture clues for effective
appearance editing, and regularized by scene depth for preserving original
geometric structure. With these novel designs, we show ReGs can produce
state-of-the-art stylization results that respect the reference texture while
embracing real-time rendering speed for free-view navigation. |
---|---|
DOI: | 10.48550/arxiv.2407.07220 |