Online Censoring for Large-Scale Regressions with Application to Streaming Big Data

• Published in: IEEE transactions on signal processing Vol. 64; no. 15; pp. 3854 - 3867
• Main Authors: Berberidis, Dimitris, Kekatos, Vassilis, Giannakis, Georgios B.
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.08.2016; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; Approximation algorithms; Big data; Budgeting; Demand; Distributed databases; Exact solutions; Inference; least squares; Linear regression; Mathematical analysis; Maximum likelihood estimation; Online; Parameter estimation; Reduction; Regression analysis; Signal processing algorithms; data reduction; Parameter estimation; censoring; support vector machines; LMS; random projections; least squares; RLS; stochastic approximation; big data
• ISSN: 1053-587X; 1941-0476
• DOI: 10.1109/TSP.2016.2546225

On par with data-intensive applications, the sheer size of modern linear regression problems creates an ever-growing demand for efficient solvers. Fortunately, a significant percentage of the data accrued can be omitted while maintaining a certain quality of statistical inference with an affordable computational budget. This work introduces means of identifying and omitting less informative observations in an online and data-adaptive fashion. Given streaming data, the related maximum-likelihood estimator is sequentially found using first- and second-order stochastic approximation algorithms. These schemes are well suited when data are inherently censored or when the aim is to save communication overhead in decentralized learning setups. In a different operational scenario, the task of joint censoring and estimation is put forth to solve large-scale linear regressions in a centralized setup. Novel online algorithms are developed enjoying simple closed-form updates and provable (non)asymptotic convergence guarantees. To attain desired censoring patterns and levels of dimensionality reduction, thresholding rules are investigated too. Numerical tests on real and synthetic datasets corroborate the efficacy of the proposed data-adaptive methods compared to data-agnostic random projection-based alternatives.