Back to results
US 11,531,578 B1Granted
Profiling and debugging for remote neural network execution
AssigneeAmazon Technologies, Inc.
InventorsRichard John Heaton, Ilya Minkin
Life of this patent
Estimating the term…
Abstract
Remote access for debugging or profiling a remotely executing neural network graph can be performed by a client using an in-band application programming interface (API). The client can provide indicator flags for debugging or profiling in an inference request sent to a remote server computer executing the neural network graph using the API. The remote server computer can collect metadata for debugging or profiling during the inference operation using the neural network graph and send it back to the client using the same API. Additionally, the metadata can be collected at various granularity levels also specified in the inference request.
Claims23
A server computer for executing a neural network graph, comprising:
- a host device configured to:
- store the neural network graph to be executed by a neural network processor coupled to the host device;
- and
- receive, using an application programming interface (API), a request message from a client computer for an inference operation to be performed by the neural network processor using the neural network graph, the request message received via a network, the request message comprising a first indicator and a second indicator, wherein the first indicator is used to enable collection of metadata during the inference operation for debugging or profiling a portion of the neural network graph, and the second indicator is used to indicate a granularity level for the collection of the metadata;
- and
- the neural network processor configured to:
- perform the inference operation using the neural network graph;
- collect the metadata during the inference operation based on the first indicator and the second indicator;
- and
- provide the collected metadata to the host device for sending to the client computer,
- wherein the host device further is configured to:
- annotate the collected metadata into the neural network graph, based on the granularity level, to generate an annotated neural network graph;
- and
- send, using the API, the annotated neural network graph to the client computer in a response message, wherein the annotated neural network graph is sent via the network.
11 more elements
A method, comprising:
- receiving, at a server computer using an application programming interface (API), a request message from a client computer to access a neural network graph of an artificial neural network, the request message comprising a first indicator and a second indicator, wherein the first indicator is used to enable collection of metadata associated with execution of the neural network graph for an operation, and wherein the second indicator is used to indicate a granularity level for collection of the metadata;
- executing, by the server computer, the artificial neural network based on the neural network graph on a neural network processing integrated circuit;
- collecting, by the server computer, the metadata representing performance of the neural network graph on the neural network processing integrated circuit based on the granularity level indicated by the second indicator;
- and
- sending, by the server computer using the application programming interface (API), the collected metadata to the client computer in a response message, the collected metadata being usable for profiling or debugging the neural network graph.
2 more elements
A system, comprising:
- a server computer configured to:
- receive, using an application programming interface (API), a request message for an inference operation to be performed by a neural network processor using a neural network graph of an artificial neural network, the request message received from a client computer, the request message comprising a first indicator and a second indicator, wherein the first indicator is used to enable collection of metadata during the inference operation for debugging or profiling a portion of the neural network graph, and wherein the second indicator is used to indicate a granularity level for collection of the metadata;
- execute the artificial neural network to perform the inference operation of the neural network graph on the neural network processor;
- collect the metadata representing performance of the neural network graph on the neural network processor based on the granularity level indicated by the second indicator;
- and
- send, using the API, the collected metadata to the client computer in a response message, the collected metadata being usable for profiling or debugging the neural network graph.
3 more elements
A non-transitory computer readable medium storing a program, wherein the program, when executed by one or more processors, causes the one or more processors to:
- send, using an application programming interface (API), a request message to a server computer for an inference operation of an artificial neural network to be performed by a neural network processor using a neural network graph, the request message comprising a first indicator and a second indicator, wherein the first indicator is used to enable collection of metadata during the inference operation for debugging or profiling a portion of the neural network graph, and the second indicator is used to indicate a granularity level for collection of the metadata;
- and
- receive, using the API, the metadata collected during execution of the artificial neural network on the neural network processor in a response message from the server computer, the metadata representing performance of the neural network graph on the neural network processor for use in profiling or debugging the neural network graph.
Dependencies are inferred from the claim text.