Virtual Occlusions Through Implicit Depth
For augmented reality (AR), it is important that virtual assets appear to 'sit among' real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer's cam...
Saved in:
Published in | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 9053 - 9064 |
---|---|
Main Authors | , , , , , , |
Format | Conference Proceeding |
Language | English |
Published |
IEEE
01.06.2023
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | For augmented reality (AR), it is important that virtual assets appear to 'sit among' real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer's camera moves. Unfortunately, small mistakes in the estimated scene depth can ruin the downstream occlusion mask, and thereby the AR illusion. Especially in real-time settings, depths inferred near boundaries or across time can be inconsistent. In this paper, we challenge the need for depth-regression as an intermediate step. We instead propose an implicit model for depth and use that to predict the occlusion mask directly. The inputs to our network are one or more color images, plus the known depths of any virtual geometry. We show how our occlusion predictions are more accurate and more temporally stable than predictions derived from traditional depth-estimation models. We obtain state-of-the-art occlusion results on the challenging ScanNetv2 dataset and superior qualitative results on real scenes. |
---|---|
AbstractList | For augmented reality (AR), it is important that virtual assets appear to 'sit among' real world objects. The virtual element should variously occlude and be occluded by real matter, based on a plausible depth ordering. This occlusion should be consistent over time as the viewer's camera moves. Unfortunately, small mistakes in the estimated scene depth can ruin the downstream occlusion mask, and thereby the AR illusion. Especially in real-time settings, depths inferred near boundaries or across time can be inconsistent. In this paper, we challenge the need for depth-regression as an intermediate step. We instead propose an implicit model for depth and use that to predict the occlusion mask directly. The inputs to our network are one or more color images, plus the known depths of any virtual geometry. We show how our occlusion predictions are more accurate and more temporally stable than predictions derived from traditional depth-estimation models. We obtain state-of-the-art occlusion results on the challenging ScanNetv2 dataset and superior qualitative results on real scenes. |
Author | Watson, Jamie Brostow, Gabriel J. Sayed, Mohamed Qureshi, Zawar Firman, Michael Vicente, Sara Aodha, Oisin Mac |
Author_xml | – sequence: 1 givenname: Jamie surname: Watson fullname: Watson, Jamie organization: Niantic – sequence: 2 givenname: Mohamed surname: Sayed fullname: Sayed, Mohamed organization: Niantic – sequence: 3 givenname: Zawar surname: Qureshi fullname: Qureshi, Zawar organization: Niantic – sequence: 4 givenname: Gabriel J. surname: Brostow fullname: Brostow, Gabriel J. organization: Niantic – sequence: 5 givenname: Sara surname: Vicente fullname: Vicente, Sara organization: Niantic – sequence: 6 givenname: Oisin Mac surname: Aodha fullname: Aodha, Oisin Mac organization: University of Edinburgh – sequence: 7 givenname: Michael surname: Firman fullname: Firman, Michael organization: Niantic |
BookMark | eNotzMtKw0AYQOFRFKw1b9BFti5S_8tcMkuJt0KhIrXbMk4mZiBNQi4L396Crg6cxXcrrtquDUKsENaIYB-Kw_uHIkN2TUC8BsiNvBCJNTZnBQxINr8UC1JGZQaMuhHJOMYvUARg2OYLcX-IwzS7Jt1538xj7Nox3ddDN3_X6ebUN9HHKX0K_VTfievKNWNI_rsUny_P--It2-5eN8XjNoukYMrQVc4za8CKpUX0JegKkVhVuvQSnAmSAbQEzxI0ITkNXGLw54cOeSlWf24MIRz7IZ7c8HNEIJDyzPwCowNDIQ |
CODEN | IEEPAD |
ContentType | Conference Proceeding |
DBID | 6IE 6IH CBEJK RIE RIO |
DOI | 10.1109/CVPR52729.2023.00874 |
DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Xplore Digital Library IEEE Proceedings Order Plans (POP) 1998-present |
DatabaseTitleList | |
Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore Digital Library url: https://proxy.k.utb.cz/login?url=https://ieeexplore.ieee.org/ sourceTypes: Publisher |
DeliveryMethod | fulltext_linktorsrc |
EISBN | 9798350301298 |
EISSN | 2575-7075 |
EndPage | 9064 |
ExternalDocumentID | 10204412 |
Genre | orig-research |
GroupedDBID | 6IE 6IH 6IL 6IN ABLEC ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP OCL RIE RIL RIO |
ID | FETCH-LOGICAL-i250t-1afac33601f34911cd06f11235f6dc40a7e4300640c3406212a603d1ec0641a13 |
IEDL.DBID | RIE |
IngestDate | Wed Jun 26 19:26:17 EDT 2024 |
IsDoiOpenAccess | false |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | true |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-LOGICAL-i250t-1afac33601f34911cd06f11235f6dc40a7e4300640c3406212a603d1ec0641a13 |
OpenAccessLink | https://www.pure.ed.ac.uk/ws/files/362466222/Virtual_Occlusions_WATSON_DOA27022023_AFV_CC_BY.pdf |
PageCount | 12 |
ParticipantIDs | ieee_primary_10204412 |
PublicationCentury | 2000 |
PublicationDate | 2023-June |
PublicationDateYYYYMMDD | 2023-06-01 |
PublicationDate_xml | – month: 06 year: 2023 text: 2023-June |
PublicationDecade | 2020 |
PublicationTitle | 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) |
PublicationTitleAbbrev | CVPR |
PublicationYear | 2023 |
Publisher | IEEE |
Publisher_xml | – name: IEEE |
SSID | ssib052007398 ssib042469789 |
Score | 2.3040473 |
Snippet | For augmented reality (AR), it is important that virtual assets appear to 'sit among' real world objects. The virtual element should variously occlude and be... |
SourceID | ieee |
SourceType | Publisher |
StartPage | 9053 |
SubjectTerms | 3D from multi-view and sensors Color Computational modeling Computer vision Geometry Lighting Measurement Predictive models |
Title | Virtual Occlusions Through Implicit Depth |
URI | https://ieeexplore.ieee.org/document/10204412 |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV09T8MwFLSgExMgivhWBhYGBzt2nXQuVAWJUqG26lY59rOoQGlVkoVfz7OTQoWExGZ58vc923f3CLk2nFuHgTjt6JxRmdmcauWA-sWBAZ3q8iAUfhqqwUQ-zjqzRqwetDAAEMhnEPti-Mu3S1P5pzLc4QlD-MYTdzdjSS3W2iwemeBFb8s63dsJpaKbNXI5zrq3venopZNgNBn7nOHe2NTT_LaSqgRM6e-T4aY1NZXkLa7KPDafv4wa_93cA9L-ke9Fo29gOiQ7UByRm-li7bUi0bMx75V_I_uIxnWSnugh0MoXZXQHq_K1TSb9-3FvQJs8CXSBAUxJuXbaCIFXKyckHl7GMuW4F8E6ZY1kOgUpwpedEYjfCFZaMWE5GKzjmotj0iqWBZyQSDPQSWZVKhlIlyVau1w4x3RulEyFOiVt38_5qrbCmG-6ePZH_TnZ82Ndc6suSKtcV3CJKF7mV2H2vgBLG5iS |
link.rule.ids | 310,311,786,790,795,796,802,27958,55109 |
linkProvider | IEEE |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwjV1NT8IwGG4MHvSkRozf7uDFQ2e7lm6cUTIUkJhBuJGuH5FggOB28df7tgMlJibelmaHfj9v2-d5XoRuFaXaQiCOGzInmCc6x1JYg93kgIBONKkXCvf6Ih3yp3FjvBarey2MMcaTz0zoPv1bvl6o0l2VwQqPCMA37Li7APSkWcm1NtOHR3DU2zJPd4ZCMWsma8Ec_H_fGg1eGxHEk6HLGu6sTR3RbyutikeV9gHqb-pTkUlmYVnkofr8ZdX47wofovqPgC8YfEPTEdox82N0N5qunFokeFHqvXS3ZB9BVqXpCTqeWD4tggezLN7qaNh-zFopXmdKwFMIYQpMpZWKMThcWcZh-1KaCEudDNYKrTiRseHMP9opBggOcCUFYZoaBWVUUnaCavPF3JyiQBIjo0SLmBPDbRJJaXNmLZG5Ejxm4gzVXTsny8oMY7Jp4vkf5TdoL8163Um303--QPuu3yum1SWqFavSXAGmF_m1H8kvkNqb6A |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2023+IEEE%2FCVF+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29&rft.atitle=Virtual+Occlusions+Through+Implicit+Depth&rft.au=Watson%2C+Jamie&rft.au=Sayed%2C+Mohamed&rft.au=Qureshi%2C+Zawar&rft.au=Brostow%2C+Gabriel+J.&rft.date=2023-06-01&rft.pub=IEEE&rft.eissn=2575-7075&rft.spage=9053&rft.epage=9064&rft_id=info:doi/10.1109%2FCVPR52729.2023.00874&rft.externalDocID=10204412 |