System and Method for Scheduling and Running Interactive Database Queries with Service Level Agreements in a Multi-Tenant Processing System

• Main Authors: Mann, Vijay, Dixit, Ankit, Tagra, Shubham, Yao, Ting, Morarka, Raunaq, Venkatesh, Rajat
• Format: Patent
• Language: English
• Published: 03.12.2020
• Subjects: CALCULATING; COMPUTING; COUNTING; ELECTRIC COMMUNICATION TECHNIQUE; ELECTRIC DIGITAL DATA PROCESSING; ELECTRICITY; PHYSICS; TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHICCOMMUNICATION

The invention is directed to systems and methods for scheduling interactive database queries from multiple tenants onto distributed query processing clusters with service level agreements (SLAs). SLAs may be provided through a combination of estimation of resources per query followed by scheduling of that query onto a cluster if enough resources are available or triggering proactive autoscaling to spawn new clusters if they are not. In some embodiments systems may include a workflow manager; a resource estimator cluster; one or more execution clusters; and one or more metastores. A workflow manager may include an active node and a passive node configured to send a query to the resource estimator cluster and receive a resource estimate. A resource estimator cluster may be in communication with the workflow manager. One or more execution clusters may be scaled by the workflow manager as part of a schedule or autoscale based on workload.