EFFICIENT STREAMING BASED LAZILY-EVALUATED MACHINE LEARNING FRAMEWORK

• Main Authors: Finley, Thomas William, Matantsev, Ivan, Luferenko, Pete, Siddiqui, Mohammad Zeeshan, Dekel, Yael, Katzenberger, Gary Shon, Eseanu, Costin, Erhardt, Eric Anthony
• Format: Patent
• Language: English
• Published: 05.11.2020
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; ELECTRIC DIGITAL DATA PROCESSING; PHYSICS

An efficient, streaming-based, lazily-evaluated machine learning (ML) framework is provided. An ML pipeline of operators produce and consume a chain of dataviews representing a computation over data. Non-materialized (e.g., virtual) views of data in dataviews permit efficient, lazy evaluation of data on demand regardless of size (e.g., in excess of main memory). Data may be materialized by DataView cursors (e.g., movable windows over rows of an input dataset or DataView). Computation and data movement may be limited to rows for active columns without processing or materializing unnecessary data. A chain of dataviews may comprise a chain of delegates that reference a chain of functions. Assembled pipelines of schematized compositions of operators may be validated and optimized with efficient execution plans. A compiled chain of functions may be optimized and executed in a single call. Dataview based ML pipelines may be developed, trained, evaluated and integrated into applications.