Dataset2Vec: learning dataset meta-features

• Published in: Data mining and knowledge discovery Vol. 35; no. 3; pp. 964 - 985
• Main Authors: Jomaa, Hadi S., Schmidt-Thieme, Lars, Grabocka, Josif
• Format: Journal Article
• Language: English
• Published: New York Springer US 01.05.2021; Springer Nature B.V
• Subjects: Artificial Intelligence; Artificial neural networks; Chemistry and Earth Sciences; Cognitive tasks; Computer Science; Data Mining and Knowledge Discovery; Datasets; Feature extraction; Information Storage and Retrieval; Machine learning; Optimization; Physics; Special Issue of the Journal Track of ECML PKDD 2021; Statistics for Engineering; Deep learning; Hyperparameter optimization; Deep sets; Meta-features; Meta-learning
• ISSN: 1384-5810; 1573-756X
• DOI: 10.1007/s10618-021-00737-9

Meta-learning, or learning to learn, is a machine learning approach that utilizes prior learning experiences to expedite the learning process on unseen tasks. As a data-driven approach, meta-learning requires meta-features that represent the primary learning tasks or datasets, and are estimated traditonally as engineered dataset statistics that require expert domain knowledge tailored for every meta-task. In this paper, first, we propose a meta-feature extractor called Dataset2Vec that combines the versatility of engineered dataset meta-features with the expressivity of meta-features learned by deep neural networks. Primary learning tasks or datasets are represented as hierarchical sets, i.e., as a set of sets, esp. as a set of predictor/target pairs, and then a DeepSet architecture is employed to regress meta-features on them. Second, we propose a novel auxiliary meta-learning task with abundant data called dataset similarity learning that aims to predict if two batches stem from the same dataset or different ones. In an experiment on a large-scale hyperparameter optimization task for 120 UCI datasets with varying schemas as a meta-learning task, we show that the meta-features of Dataset2Vec outperform the expert engineered meta-features and thus demonstrate the usefulness of learned meta-features for datasets with varying schemas for the first time.