Engineering a fast online persistent suffix tree construction

• Published in: Proceedings. 20th International Conference on Data Engineering pp. 720 - 731
• Main Authors: Bedathur, S.J., Haritsa, J.R.
• Format: Conference Proceeding
• Language: English
• Published: Los Alamitos CA IEEE 2004; IEEE Computer Society
• Subjects: Applied sciences; Bioinformatics; Computer science; control theory; systems; Costs; Database systems; DNA; Exact sciences and technology; Genetics; Genomics; Indexes; Indexing; Information systems. Data bases; Memory organisation. Data processing; Proteins; Sequences; Software; Tree structured method; Suffix; DNA; Budget; Tree structure; Buffer system; Genome; Protein
• ISBN: 9780769520650; 0769520650
• ISSN: 1063-6382; 2375-026X
• DOI: 10.1109/ICDE.2004.1320040

Online persistent suffix tree construction has been considered impractical due to its excessive I/O costs. However, these prior studies have not taken into account the effects of the buffer management policy and the internal node structure of the suffix tree on I/O behavior of construction and subsequent retrievals over the tree. We study these two issues in detail in the context of large genomic DNA and protein sequences. In particular, we make the following contributions: (i) a novel, low-overhead buffering policy called TOP-Q which improves the on-disk behavior of suffix tree construction and subsequent retrievals, and (ii) empirical evidence that the space efficient linked-list representation of suffix tree nodes provides significantly inferior performance when compared to the array representation. These results demonstrate that a careful choice of implementation strategies can make online persistent suffix tree construction considerably more scalable - in terms of length of sequences indexed with a fixed memory budget, than currently perceived.