SYSTEM AND METHOD FOR PARALLELIZING LORAS BY MAXIMIZING GPU UTILIZATION

Abstract

One example method includes receiving multiple LoRA (low rank adaptor) models, batching the LoRA models together to generate one or more batches of the LoRA models, creating a respective queue for each of the batches of the LoRA models, calling the LoRA models in a sequence in which the LoRA models were batched, and using only a single GPU (graphics processing unit), performing simultaneous parallel inferencing on all of the LoRA models.

CPC Classifications