US 11,694,066 B2 · Machine learning runtime library for neural network acceleration

Abstract

Embodiments herein describe techniques for interfacing a neural network application with a neural network accelerator using a library. The neural network application may execute on a host computing system while the neural network accelerator executes on a massively parallel hardware system, e.g., a FPGA. The library operates a pipeline for submitting the tasks received from the neural network application to the neural network accelerator. In one embodiment, the pipeline includes a pre-processing stage, an FPGA execution stage, and a post-processing stage which each correspond to different threads. When receiving a task from the neural network application, the library generates a packet that includes the information required for the different stages in the pipeline to perform the tasks. Because the stages correspond to different threads, the library can process multiple packets in parallel which can increase the utilization of the neural network accelerator on the hardware system.

Claims20

Dependent claims

A method for pipelining tasks submitted to a neural network accelerator, the method comprising:

establishing a pipeline comprising multiple stages in a library, wherein the pipeline and the library execute on a host;
receiving, at the library executing on the host, a first task from a neural network application to be processed by the neural network accelerator;
generating, using one or more computer processors, a packet that contains information used by the multiple stages in the pipeline operating in the library, wherein the library is an adapter between the neural network application executing on the host and the neural network accelerator executing on a hardware system;
processing the packet at the multiple stages, wherein at least one of the multiple stages performs a call to the neural network accelerator executing on the hardware system, and wherein the pipeline processes at least one other packet corresponding to a second task in parallel with processing the packet;
and
returning results of processing the packet using the pipeline to the neural network application.

3 more elements

A computing system, comprising:

a processor;
and
a memory comprising a library that is an adapter between a neural network application and a neural network accelerator executing on a hardware system, wherein the library, when executed by the processor performs an operation comprising:
receiving a first task from the neural network application to be processed by the neural network accelerator;
generating, using one or more computer processors, a packet that contains information used by multiple stages in a pipeline within the library;
processing the packet at the multiple stages, wherein at least one of the multiple stages performs a call to the neural network accelerator executing on the hardware system, and wherein the pipeline processes at least one other packet corresponding to a second task in parallel with processing the packet;
and
returning results of processing the packet using the pipeline to the neural network application.

5 more elements

Dependencies are inferred from the claim text.

Machine learning runtime library for neural network acceleration

Life of this patent

Machine learning runtime library for neural network acceleration

Life of this patent

Abstract

Claims20