Systems and methods for performing knowledge distillation

• Main Authors: Duerig, Thomas J, Rudkin, Scott Alexander, Wang, Hongsheng
• Format: Patent
• Language: English
• Published: 17.10.2023
• Subjects: CALCULATING; COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS; COMPUTING; COUNTING; PHYSICS

The present disclosure is directed to methods and systems for knowledge distillation. Implementations of the disclosure can include executing the following actions using one or more computing devices: obtaining an initial training dataset including multiple training examples; determining sets of outputs by performing inference on the training examples with a group of pre-trained machine-learned models that have been trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a performance of each pretrained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using the distillation training dataset.