Magic Layouts: Structural Prior for Component Detection in User Interface Designs

We present Magic Layouts; a method for parsing screen-shots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar....

Full description

Saved in:
Bibliographic Details
Published in2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 15804 - 15813
Main Authors Manandhar, Dipu, Jin, Hailin, Collomosse, John
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.06.2021
Subjects
Layout
Location awareness
Object detection
Performance gain
Prototypes
User interfaces
Online AccessGet full text

Cover

More Information
Summary:We present Magic Layouts; a method for parsing screen-shots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screen-shots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.
ISSN:2575-7075
DOI:10.1109/CVPR46437.2021.01555