Regression Cubes with Lossless Compression and Aggregation

• Published in: IEEE transactions on knowledge and data engineering Vol. 18; no. 12; pp. 1585 - 1599
• Main Authors: Chen, Y., Dong, G., Han, J., Pei, J., Wah, B.W., Jianyong Wang
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01.12.2006; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Agglomeration; Aggregation; Applied sciences; compression; Computational efficiency; Computer science; control theory; systems; Cubes; Data analysis; data cubes; Data processing. List processing. Character string processing; Delay; Distributed computing; Engines; Exact sciences and technology; Extraterrestrial measurements; Filtering; Filtration; Loss measurement; Mathematical models; Memory organisation. Data processing; Multidimensional data; Multidimensional systems; OLAP; Online analytical processing; Predictive models; Regression; Size measurement; Software; Studies; Data analysis; Statistical analysis; On line; Filtering; Data compression; Lossless compression; Data processing; OLAP; Modeling; Time response; data cubes; Statistical data; Statistical regression; Aggregation; Multidimensional analysis; Multidimensional database; Statistical computation; Response time; Cube; compression; Time analysis; User behavior; Data warehouse
• ISSN: 1041-4347; 1558-2191
• DOI: 10.1109/TKDE.2006.196

As OLAP engines are widely used to support multidimensional data analysis, it is desirable to support in data cubes advanced statistical measures, such as regression and filtering, in addition to the traditional simple measures such as count and average. Such new measures allow users to model, smooth, and predict the trends and patterns of data. Existing algorithms for simple distributive and algebraic measures are inadequate for efficient computation of statistical measures in a multidimensional space. In this paper, we propose a fundamentally new class of measures, compressible measures, in order to support efficient computation of the statistical models. For compressible measures, we compress each cell into an auxiliary matrix with a size independent of the number of tuples. We can then compute the statistical measures for any data cell from the compressed data of the lower-level cells without accessing the raw data. Time- and space-efficient lossless aggregation formulae are derived for regression and filtering measures. Our analytical and experimental studies show that the resulting system, regression cube, substantially reduces the memory usage and the overall response time for statistical analysis of multidimensional data