LW-FQZip 2: a parallelized reference-based compression of FASTQ files

• Published in: BMC bioinformatics Vol. 18; no. 1; p. 179
• Main Authors: Huang, Zhi-An, Wen, Zhenkun, Deng, Qingjin, Chu, Ying, Sun, Yiwen, Zhu, Zexuan
• Format: Journal Article
• Language: English
• Published: England BioMed Central 20.03.2017
• Subjects: Acids; Algorithms; Alignment; Archives & records; Archiving; Arithmetic; Arithmetic coding; Assembling; Assembly; Bioinformatics; Compression; Computer programs; Conserved sequence; Data compression; Data Compression - methods; Data transfer (computers); Data transmission; Deoxyribonucleic acid; DNA; DNA methylation; DNA sequencing; Filters; Fragmentation; Gene mapping; Genetic transformation; Genomes; High-Throughput Nucleotide Sequencing - methods; Homology; Mapping; Matching; Mathematical models; Metadata; Methods; Nucleotide sequence; Nucleotides; Palindromes; Porosity; Probability theory; Quality; Redundancy; Sequence Alignment - methods; Sequence Analysis, DNA - methods; Silver; Software; Statistical analysis; Streams; Sun; Weight reduction; Sequencing data compression; Reference- based compression; High-throughput sequencing; Sequence alignment
• ISSN: 1471-2105; 1471-2105
• DOI: 10.1186/s12859-017-1588-x

The rapid progress of high-throughput DNA sequencing techniques has dramatically reduced the costs of whole genome sequencing, which leads to revolutionary advances in gene industry. The explosively increasing volume of raw data outpaces the decreasing disk cost and the storage of huge sequencing data has become a bottleneck of downstream analyses. Data compression is considered as a solution to reduce the dependency on storage. Efficient sequencing data compression methods are highly demanded. In this article, we present a lossless reference-based compression method namely LW-FQZip 2 targeted at FASTQ files. LW-FQZip 2 is improved from LW-FQZip 1 by introducing more efficient coding scheme and parallelism. Particularly, LW-FQZip 2 is equipped with a light-weight mapping model, bitwise prediction by partial matching model, arithmetic coding, and multi-threading parallelism. LW-FQZip 2 is evaluated on both short-read and long-read data generated from various sequencing platforms. The experimental results show that LW-FQZip 2 is able to obtain promising compression ratios at reasonable time and memory space costs. The competence enables LW-FQZip 2 to serve as a candidate tool for archival or space-sensitive applications of high-throughput DNA sequencing data. LW-FQZip 2 is freely available at http://csse.szu.edu.cn/staff/zhuzx/LWFQZip2 and https://github.com/Zhuzxlab/LW-FQZip2 .