System for routing machine learning model inferences

A machine learning model inference routing system in a machine learning service is described herein. The machine learning model inference routing system includes load balancer(s), network traffic router(s), an endpoint registry, and a feedback processing system that collectively allow the machine le...

Full description

Saved in:
Bibliographic Details
Main Authors Stefani, Stefano, Dirac, Leo Parker, Goodhart, Taylor
Format Patent
LanguageEnglish
Published 09.11.2021
Subjects
CALCULATING
COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
COMPUTING
COUNTING
ELECTRIC COMMUNICATION TECHNIQUE
ELECTRICITY
PHYSICS
TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHICCOMMUNICATION
Online AccessGet full text

Cover

More Information
Summary:A machine learning model inference routing system in a machine learning service is described herein. The machine learning model inference routing system includes load balancer(s), network traffic router(s), an endpoint registry, and a feedback processing system that collectively allow the machine learning model inference routing system to adjust the routing of inferences based on machine learning model accuracy, demand, and/or the like. In addition, the arrangement of components in the machine learning model inference routing system enables the machine learning service to perform shadow testing, support ensemble machine learning models, and/or improve existing machine learning models using feedback data.
Bibliography:Application Number: US201715821564