SLOPE for Sparse Linear Regression: Asymptotics and Optimal Regularization

• Published in: IEEE transactions on information theory Vol. 68; no. 11; pp. 7627 - 7664
• Main Authors: Hu, Hong, Lu, Yue M.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.11.2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Asymptotic properties; asymptotics; Computational geometry; Convexity; Estimation; Input variables; Limiting; Linear regression; optimal regularization; Optimization; Regularization; SLOPE estimator; Task analysis; Visualization
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2022.3188753

In sparse linear regression, the SLOPE estimator generalizes LASSO by penalizing different coordinates of the estimate according to their magnitudes. In this paper, we present a precise performance characterization of SLOPE under the i.i.d. Gaussian design. We focus on the asymptotic regime where the number of unknown parameters grows in proportion to the number of observations. Our asymptotic characterization enables us to derive the fundamental limits of SLOPE in both estimation and variable selection settings. We also provide a computationally feasible way to optimally design the regularizing sequences to reach the aforementioned fundamental limits. In both settings, we show that the optimal design problem can be formulated as certain infinite-dimensional convex optimization problems that have efficient and accurate finite-dimensional approximations. Numerical simulations verify all our asymptotic predictions. They demonstrate the superiority of our optimal regularizing sequences over other designs used in the existing literature.