PROCESSING PARALLELISM FOR MACHINE LEARNING MODEL TRAINING

Abstract

A processing system schedules parallel training of different instances of a machine learning model (MLM) based on a number of microbatches associated with training the machine learning model. The number of microbatches, along with the time required to complete a forward and backward pass of the MLM per microbatch, indicates the position, in time, of one or more expected idle cycles of a processing unit during training of a first instance of the MLM. A scheduler of the processing system schedules a second instance of the MLM during the one or more expected idle cycles.

CPC Classifications