Multimodal Search on Iconclass Using Vision-Language Pre-Trained Models

Terminology sources, such as controlled vocabularies, thesauri and classification systems, play a key role in digitizing cultural heritage. However, Information Retrieval (IR) systems that allow to query and explore these lexical resources often lack an adequate representation of the semantics behin...

Full description

Saved in:
Bibliographic Details
Published in2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL) pp. 285 - 287
Main Authors Santini, Cristian, Posthumus, Etienne, Tietz, Tabea, Tan, Mary Ann, Bruns, Oleksandra, Sack, Harald
Format Conference Proceeding
LanguageEnglish
Published IEEE 01.06.2023
Subjects
art history
classification systems
information retrieval
Libraries
multimodal search
Search engines
Semantics
Terminology
vision-language models
Visualization
Vocabulary
Online AccessGet full text

Cover

More Information
Summary:Terminology sources, such as controlled vocabularies, thesauri and classification systems, play a key role in digitizing cultural heritage. However, Information Retrieval (IR) systems that allow to query and explore these lexical resources often lack an adequate representation of the semantics behind the user's search, which can be conveyed through multiple expression modalities (e.g., images, keywords or textual descriptions). This paper presents the implementation of a new search engine for one of the most widely used iconography classification system, Iconclass. The novelty of this system is the use of a pre-trained vision-language model, namely CLIP, to retrieve and explore Iconclass concepts using visual or textual queries.
ISSN:2575-8152
DOI:10.1109/JCDL57899.2023.00061