Human-VDM: Learning Single-Image 3D Human Gaussian Splatting from Video Diffusion Models
Generating lifelike 3D humans from a single RGB image remains a challenging task in computer vision, as it requires accurate modeling of geometry, high-quality texture, and plausible unseen parts. Existing methods typically use multi-view diffusion models for 3D generation, but they often face incon...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
04.09.2024
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Generating lifelike 3D humans from a single RGB image remains a challenging task in computer vision, as it requires accurate modeling of geometry, high-quality texture, and plausible unseen parts. Existing methods typically use multi-view diffusion models for 3D generation, but they often face inconsistent view issues, which hinder high-quality 3D human generation. To address this, we propose Human-VDM, a novel method for generating 3D human from a single RGB image using Video Diffusion Models. Human-VDM provides temporally consistent views for 3D human generation using Gaussian Splatting. It consists of three modules: a view-consistent human video diffusion module, a video augmentation module, and a Gaussian Splatting module. First, a single image is fed into a human video diffusion module to generate a coherent human video. Next, the video augmentation module applies super-resolution and video interpolation to enhance the textures and geometric smoothness of the generated video. Finally, the 3D Human Gaussian Splatting module learns lifelike humans under the guidance of these high-resolution and view-consistent images. Experiments demonstrate that Human-VDM achieves high-quality 3D human from a single image, outperforming state-of-the-art methods in both generation quality and quantity. Project page: https://human-vdm.github.io/Human-VDM/ |
---|---|
AbstractList | Generating lifelike 3D humans from a single RGB image remains a challenging task in computer vision, as it requires accurate modeling of geometry, high-quality texture, and plausible unseen parts. Existing methods typically use multi-view diffusion models for 3D generation, but they often face inconsistent view issues, which hinder high-quality 3D human generation. To address this, we propose Human-VDM, a novel method for generating 3D human from a single RGB image using Video Diffusion Models. Human-VDM provides temporally consistent views for 3D human generation using Gaussian Splatting. It consists of three modules: a view-consistent human video diffusion module, a video augmentation module, and a Gaussian Splatting module. First, a single image is fed into a human video diffusion module to generate a coherent human video. Next, the video augmentation module applies super-resolution and video interpolation to enhance the textures and geometric smoothness of the generated video. Finally, the 3D Human Gaussian Splatting module learns lifelike humans under the guidance of these high-resolution and view-consistent images. Experiments demonstrate that Human-VDM achieves high-quality 3D human from a single image, outperforming state-of-the-art methods in both generation quality and quantity. Project page: https://human-vdm.github.io/Human-VDM/ |
Author | Dong, Haoye Wu, Hefeng Chharia, Aviral Liu, Zhibin |
Author_xml | – sequence: 1 givenname: Zhibin surname: Liu fullname: Liu, Zhibin – sequence: 2 givenname: Haoye surname: Dong fullname: Dong, Haoye – sequence: 3 givenname: Aviral surname: Chharia fullname: Chharia, Aviral – sequence: 4 givenname: Hefeng surname: Wu fullname: Wu, Hefeng |
BookMark | eNqNykELgjAABeARBVn5HwadB3NLy65ZGeTJkG4ycJPJ3Gxz_z-LfkCX9x68bwXm2mg-AwGhNEKHHSFLEDrXYYxJsidxTAPwzH3PNKqy4gjvnFktdQvLKRRHt561HNIMfg28Mu-cnEY5KDaOHyis6WElG25gJoXwThoNC9Nw5TZgIZhyPPz1Gmwv58cpR4M1L8_dWHfGWz1dNY0wTtMkjTD9T70BIopCJw |
ContentType | Paper |
Copyright | 2024. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2024. This work is published under http://creativecommons.org/licenses/by-nc-sa/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central ProQuest Central Essentials ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection ProQuest Engineering Collection Engineering Database Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_31009969103 |
IEDL.DBID | 8FG |
IngestDate | Wed Sep 25 02:10:54 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_31009969103 |
OpenAccessLink | https://www.proquest.com/docview/3100996910/abstract/?pq-origsite=%requestingapplication% |
PQID | 3100996910 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_3100996910 |
PublicationCentury | 2000 |
PublicationDate | 20240904 |
PublicationDateYYYYMMDD | 2024-09-04 |
PublicationDate_xml | – month: 09 year: 2024 text: 20240904 day: 04 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2024 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.5518334 |
SecondaryResourceType | preprint |
Snippet | Generating lifelike 3D humans from a single RGB image remains a challenging task in computer vision, as it requires accurate modeling of geometry, high-quality... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Computer vision Image enhancement Image quality Image resolution Modules Smoothness |
Title | Human-VDM: Learning Single-Image 3D Human Gaussian Splatting from Video Diffusion Models |
URI | https://www.proquest.com/docview/3100996910/abstract/ |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT4NAEJ7UEhNvPuOjNpvodYPSpVgvJgoUTWgaqw23hsdimiBggau_3ZkNaOKhl01YCCwb5vUxMx_AdZhYUkiUb4G-BycPnEdjy-ByIoWQIhKm6t3pz8beu3gJzKAHXlcLQ2mVnU5UijopYsLIdQKi0TdH66aHEaEAca0_lF-c-KPoP2tLprED2i31xKOacXf6i7YY-HyTCJD_KVxlRdx90OZhKTcH0JP5Ieyq5Mu4OoJAIel8afv3rO13-sEWOGSSP3-iwLORzdQ1bBo2FZU9skWZhSplmVGBCFuuE1kwe52mDcFfjCjOsuoYrlzn7cnj3XJW7adTrf5edHQC_bzI5Smw8CZKYwNlzYonuLURRk1pLCnWMmjm7gwG2-50vv30BewZaKtV6pQYQL_eNPISbW0dDdU2DkF7dGbzVzzyv50fExiImA |
link.rule.ids | 786,790,12792,21416,33408,33779,43635,43840 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT8MwDLZgE4IbT_EYEAmuEahNV8aFA6XrYJ2QNqbeqjZ1p0mFlXX9_9hRBxKHXXJIqrSJan_2FzsGuE0yFxWSfCuyPSRb4DLtupbEHiqFKlWOubszHHWDD_UaOVFDuFVNWOVaJxpFnS00c-R3TESTbU7o9lR-S64axaerTQmNbWgrm6CTM8X9_i_HYtFbHS57_E_NGuzw96H9npS4PIAt_DqEHRNyqasjiAx_Lqde-CiaW05nYkxNgXLwSWIubE-YZ0Q_qStOdhTjskhMoLLgtBAxnWe4EN48z2smvQQXNiuqY7jxXybPgVx_Ttz8MFX8tzz7BFrk-eMpiOQ-zbVFEubqHm1oSr5SrpE9LIt7Hs6gs2mm883D17AbTMJhPByM3i5gzyK0NsFTqgOt1bLGS0LbVXpltvQHh0GE-w |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Human-VDM%3A+Learning+Single-Image+3D+Human+Gaussian+Splatting+from+Video+Diffusion+Models&rft.jtitle=arXiv.org&rft.au=Liu%2C+Zhibin&rft.au=Dong%2C+Haoye&rft.au=Chharia%2C+Aviral&rft.au=Wu%2C+Hefeng&rft.date=2024-09-04&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |