Asynchronous, scalable and customizable model serving over pipes
AssigneeFreshworks Inc.
InventorArrman Anicket SAHA
Life of this patent
Estimating the term…
Abstract
A method for serving machine learning (ML) model over at least two pipes includes defining a function that receives at least two process pipes as arguments, and defining another class. The other class is used for creating and storing future objects and setting the result from another thread. The method also includes releasing a semaphore to avoid overloading of a second pipe once the data is sent to the other thread, and writing an asynchronous function to a first pipe or an input pipe in order to send the data to the other thread.
Claims20
A computer-implemented method for serving machine learning (ML) model over at least two pipes, the method comprising:
- defining a function that receives at least two process pipes as arguments;
- defining another class, wherein the other class is used for creating and storing future objects and setting the result from another thread.
- releasing a semaphore to avoid overloading of a second pipe once the data is sent to the other thread;
- and
- writing an asynchronous function to a first pipe or an input pipe in order to send the data to the other thread.
A computer-implemented method for processing batched data using a plurality of pipelines, comprising:
- receiving, from a first process, data comprising a plurality of text, each of which corresponding to one of a plurality of models;
- batching, at a second process, the data;
- sending, at the second process, the batched data to a corresponding one of a plurality of pipelines;
- executing, at the second process, each of the corresponding ones of the plurality of pipelines for the batch data;
- and
- returning, from the second process, to the first process a final result from the executed corresponding ones of the plurality of the pipelines.
Dependencies are inferred from the claim text.