DATA: Differentiable ArchiTecture Approximation With Distribution Guided Sampling

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 43; no. 9; pp. 2905 - 2920
• Main Authors: Zhang, Xinbang, Chang, Jianlong, Guo, Yiwen, Meng, Gaofeng, Xiang, Shiming, Lin, Zhouchen, Pan, Chunhong
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.09.2021; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Approximation; Back propagation; Binary codes; Bridges; Clustering; Computer architecture; distribution guided sampling; ensemble gumbel-softmax; Estimation; Image classification; Image segmentation; Learning; Mathematical analysis; Modelling; Neural architecture search(NAS); Optimization; Sampling; Search problems; Searching; Task analysis
• ISSN: 0162-8828; 1939-3539; 2160-9292
• DOI: 10.1109/TPAMI.2020.3020315

Neural architecture search (NAS) is inherently subject to the gap of architectures during searching and validating. To bridge this gap effectively, we develop Differentiable ArchiTecture Approximation (DATA) with Ensemble Gumbel-Softmax (EGS) estimator and Architecture Distribution Constraint (ADC) to automatically approximate architectures during searching and validating in a differentiable manner. Technically, the EGS estimator consists of a group of Gumbel-Softmax estimators, which is capable of converting probability vectors to binary codes and passing gradients reversely, reducing the estimation bias in a differentiable way. To narrow the distribution gap between sampled architectures and supernet, further, the ADC is introduced to reduce the variance of sampling during searching. Benefiting from such modeling, architecture probabilities and network weights in the NAS model can be jointly optimized with the standard back-propagation, yielding an end-to-end learning mechanism for searching deep neural architectures in an extended search space. Conclusively, in the validating process, a high-performance architecture that approaches to the learned one during searching is readily built. Extensive experiments on various tasks including image classification, few-shot learning, unsupervised clustering, semantic segmentation and language modeling strongly demonstrate that DATA is capable of discovering high-performance architectures while guaranteeing the required efficiency. Code is available at https://github.com/XinbangZhang/DATA-NAS .