Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction
Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necess...
Saved in:
Published in | arXiv.org |
---|---|
Main Authors | , , , , , , |
Format | Paper |
Language | English |
Published |
Ithaca
Cornell University Library, arXiv.org
22.05.2020
|
Subjects | |
Online Access | Get full text |
Cover
Loading…
Abstract | Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necessity of exploring multi-frequency analysis to deal with the two problems. Inspired by the frequency band decomposition characteristic of Human Vision System (HVS), we propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner. Specifically, the multi-level spatial discrete wavelet transform decomposes each video frame into anisotropic sub-bands with multiple frequencies, helping to enrich structural information and reserve fine details. On the other hand, multi-level temporal discrete wavelet transform which operates on time axis decomposes the frame sequence into sub-band groups of different frequencies to accurately capture multi-frequency motions under a fixed frame rate. Extensive experiments on diverse datasets demonstrate that our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works. |
---|---|
AbstractList | Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two major problems for current predictive models, which lead to image distortion and temporal inconsistency. In this paper, we point out the necessity of exploring multi-frequency analysis to deal with the two problems. Inspired by the frequency band decomposition characteristic of Human Vision System (HVS), we propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner. Specifically, the multi-level spatial discrete wavelet transform decomposes each video frame into anisotropic sub-bands with multiple frequencies, helping to enrich structural information and reserve fine details. On the other hand, multi-level temporal discrete wavelet transform which operates on time axis decomposes the frame sequence into sub-band groups of different frequencies to accurately capture multi-frequency motions under a fixed frame rate. Extensive experiments on diverse datasets demonstrate that our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works. |
Author | Tang, Qiankun Li, Xiaowei Niu, Jingyu Han, Yinhe Shi, Zhiping Jin, Beibei Hu, Yu |
Author_xml | – sequence: 1 givenname: Beibei surname: Jin fullname: Jin, Beibei – sequence: 2 givenname: Yu surname: Hu fullname: Hu, Yu – sequence: 3 givenname: Qiankun surname: Tang fullname: Tang, Qiankun – sequence: 4 givenname: Jingyu surname: Niu fullname: Niu, Jingyu – sequence: 5 givenname: Zhiping surname: Shi fullname: Shi, Zhiping – sequence: 6 givenname: Yinhe surname: Han fullname: Han, Yinhe – sequence: 7 givenname: Xiaowei surname: Li fullname: Li, Xiaowei |
BookMark | eNqNjbEKwjAUAIMoWLX_EHAOxMTWriKWLoKguJbQRn0l5tUkBfv3FtHd6ZY7bkbGFq0ekUhIuWLZWogpib1vOOci3YgkkRHB_as16MDe6KlVAZRhZ_1o0SlDD50JwHKnn522VU-3Vpneg6dXdLSA253lUGsDoafK1vTXsR3awQqf5jIYSI9O11AFQLsgk6syXsdfzsky3593BWsdDhsfygY7N4x8KWQqk4ynGZf_WW9wjUy1 |
ContentType | Paper |
Copyright | 2020. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
Copyright_xml | – notice: 2020. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials AUTh Library subscriptions: ProQuest Central Technology Collection ProQuest One Community College ProQuest Central SciTech Premium Collection (Proquest) (PQ_SDU_P3) ProQuest Engineering Collection ProQuest Engineering Database Publicly Available Content Database ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Academic ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest One Academic Engineering Collection |
DatabaseTitleList | Publicly Available Content Database |
Database_xml | – sequence: 1 dbid: 8FG name: ProQuest Technology Collection url: https://search.proquest.com/technologycollection1 sourceTypes: Aggregation Database |
DeliveryMethod | fulltext_linktorsrc |
Discipline | Physics |
EISSN | 2331-8422 |
Genre | Working Paper/Pre-Print |
GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PIMPY PQEST PQQKQ PQUKI PRINS PTHSS |
ID | FETCH-proquest_journals_23635806803 |
IEDL.DBID | BENPR |
IngestDate | Tue Sep 24 21:09:27 EDT 2024 |
IsOpenAccess | true |
IsPeerReviewed | false |
IsScholarly | false |
Language | English |
LinkModel | DirectLink |
MergedId | FETCHMERGED-proquest_journals_23635806803 |
OpenAccessLink | https://www.proquest.com/docview/2363580680/abstract/?pq-origsite=%requestingapplication% |
PQID | 2363580680 |
PQPubID | 2050157 |
ParticipantIDs | proquest_journals_2363580680 |
PublicationCentury | 2000 |
PublicationDate | 20200522 |
PublicationDateYYYYMMDD | 2020-05-22 |
PublicationDate_xml | – month: 05 year: 2020 text: 20200522 day: 22 |
PublicationDecade | 2020 |
PublicationPlace | Ithaca |
PublicationPlace_xml | – name: Ithaca |
PublicationTitle | arXiv.org |
PublicationYear | 2020 |
Publisher | Cornell University Library, arXiv.org |
Publisher_xml | – name: Cornell University Library, arXiv.org |
SSID | ssj0002672553 |
Score | 3.2753155 |
SecondaryResourceType | preprint |
Snippet | Video prediction is a pixel-wise dense prediction task to infer future frames based on past frames. Missing appearance details and motion blur are still two... |
SourceID | proquest |
SourceType | Aggregation Database |
SubjectTerms | Accuracy Banded structure Consistency Decomposition Discrete Wavelet Transform Frequencies Frequency analysis Prediction models Vision systems Wavelet analysis Wavelet transforms |
Title | Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity and Temporal-Consistency Video Prediction |
URI | https://www.proquest.com/docview/2363580680/abstract/ |
hasFullText | 1 |
inHoldings | 1 |
isFullTextHit | |
isPrint | |
link | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1La8MwDBZtw2C3PdmjK4btapI4z54GG83CoCWMbvRWklg5jaRL0sMu--2z3KQ7DHo0toyxhSRLnySAB4FyGngy45gLyV1hKTmIBfIgTLNCJw4ISnCeL_z43X1deasBxH0uDMEqe5moBbWscvKRm8LxKWLnh5aZZuQFyFvzcfPFqX8UxVm7ZhpDMITtUsDWeJotkre9v0X4gbKenX8iV-uR6ASMJN1gfQoDLM_gSMMv8-Ycqj0UjlGPYMUTfLmrGfXJdIosj-od5vmb9WVEmDI3GcE0eESlqpQ1zdJSsp6O61acTatpPtSKiiU1RWXoJS7gPpotn2Pen3Ld8VSz_rsB5xJGZVXiFbCwsLEIbJvKbLlWGmSOM_WEVFYfWnaI3jWMD-10c3j6Fo4F_S8tjwsxhlFbb_FOKeE2m8AwjF4m3S2r0fxn9gtc-JQw |
link.rule.ids | 786,790,12792,21416,33408,33779,43635,43840 |
linkProvider | ProQuest |
linkToHtml | http://utb.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwfV1LT8MwDLZgE4IbT_EYEAmuEW36SHfigCgFtmmHgnar2sY9oXZry4F_T5y144C0s-MoiSzbsT_bAPcC1Vh6KuOYC8VdYWk9iAVyGaRZYQoHBBU4T2d-9OG-LbxFF3BrOlhlrxONolZVTjHyB-H4lLHzA-txueI0NYqyq90IjV0YukSnSvHwZRNjEb7UHrPzT80a2xEewnCeLrE-gh0sj2HPQC7z5gSqDfyN0VxgLQc8XveJ-mKmLJaH9Rrn_MP61iFMu5iMoBk8pPZU2oNmaalYz8fN-M2mNTyfekXF5jVlYuj1T-EufI6fIt6fMunkqEn-bu2cwaCsSjwHFhQ2FtK2qbWWa6Uyc5yxJ5T29NCyA_QuYLRtp8vt5FvYj-LpJJm8zt6v4EDQ_9LyuBAjGLT1N15rI9xmN-alfwETI4_m |
openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Exploring+Spatial-Temporal+Multi-Frequency+Analysis+for+High-Fidelity+and+Temporal-Consistency+Video+Prediction&rft.jtitle=arXiv.org&rft.au=Jin%2C+Beibei&rft.au=Hu%2C+Yu&rft.au=Tang%2C+Qiankun&rft.au=Niu%2C+Jingyu&rft.date=2020-05-22&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422 |