Distributed PARAFAC Decomposition Method Based on In-memory Big Data System

• Published in: Database Systems for Advanced Applications Vol. 11448; pp. 292 - 295
• Main Authors: Yang, Hye-Kyung, Yong, Hwan-Seung
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2019; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Apache Spark; Distributed PARAFAC decomposition
• ISBN: 3030185893; 9783030185893
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-030-18590-9_31

We propose IM-PARAFAC, a PARAFAC tensor decomposition method that enables rapid processing of large scalable tensors in Apache Spark for distributed in-memory big data management systems. We consider the memory overflow that occurs when processing large amounts of data because of running on in-memory. Therefore, the proposed method, IM-PARAFAC, is capable of dividing and decomposing large input tensors. It can handle large tensors even in small, distributed environments. The experimental results indicate that the proposed IM-PARAFAC enables handling of large tensors and reduces the execution time.