Distributed storage and parallel calculation-based power grid data quality detection method

• Main Authors: LONG QINGLIN, CHEN CHENGZHI, LIANG GUOHUI, HUANG YIHUA, GU RONG, YANG BINCHENG
• Format: Patent
• Language: English
• Published: 04.03.2015
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

The invention discloses a distributed storage and parallel calculation-based power grid data quality detection method, which comprises the following steps of storing an original data record by adopting an HBase; establishing a query index for a field related to a checking rule by adopting the HBase; establishing a timestamp index for the original data record so as to provide support for incremental data quality checking and small-time granularity data quality checking by adopting the HBase; storing an auxiliary index file and an operation log file of the data record so as to rapidly load checking data and improve checking performance during total historical data quality checking by adopting an HDFS (hadoop distributed file system); performing MapReduce-based checking rule parallel processing to improve the checking performance. According to the method, the problems of poor extensibility, long checking time delay and low system cost performance of a conventional relational database system-based power grid data quality detection method are solved.