# US 11,948,053 B2 · Inferencer graph for implementing machine learning model topology

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,948,053 B2
- Document ID: US11948053B2
- Status: Granted
- Kind: B2
- Application number: 17/025,871
- Filed: 2020-09-18
- Published: 2022-03-24
- Granted: 2024-04-02
- Earliest claimed: 2020-09-18
- Assignees: Megh Computing, Inc.
- Applicants: Megh Computing, Inc.
- Inventors: Duncan Moss
- Web page: https://patentgrep.com/patent/US11948053B2

## CPC

- G06N 3/063: using electronic means
- G06N 20/00: Machine learning
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06Q 10/06315: {Needs-based resource requirements planning or analysis}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A computing system, including a processor configured to, at development time, receive a machine learning model topology including a plurality of layers. The processor may be further configured to generate an internal representation graph of the machine learning model topology. The internal representation graph may include a plurality of internal representation layers. By performing one or more modifications to the internal representation graph, the processor may be further configured to generate an inferencer graph including a plurality of inferencer layer blocks. Each inferencer layer block may indicate an input buffer size, a logic function, and an output buffer size. At deployment time, the processor may be further configured to transmit, to a plurality of processing devices, instructions to implement the machine learning model topology with the respective input buffer sizes, logic functions, and output buffer sizes selected for the plurality of inferencer layer blocks of the inferencer graph.

## Claims

### Claim 1

A computing system comprising:
a processor configured to:
at development time:
receive a machine learning model topology including a plurality of layers; and
generate an internal representation graph of the machine learning model topology, wherein the internal representation graph includes a plurality of internal representation layers; and
at least in part by performing one or more modifications to the internal representation graph, generate an inferencer graph including a plurality of inferencer layer blocks, wherein each inferencer layer block indicates an input buffer size, a logic function, and an output buffer size; and
at deployment time, transmit, to a plurality of processing devices, instructions to implement the machine learning model topology with the respective input buffer sizes, logic functions, and output buffer sizes indicated in the plurality of inferencer layer blocks of the inferencer graph.

### Claim 2 (depends on 1)

The computing system of claim 1, wherein, at development time, the processor is further configured to:
receive a respective plurality of performance targets for the plurality of layers of the machine learning model topology; and
generate the inferencer graph such that the plurality of inferencer layer blocks meet the respective plurality of performance targets.

### Claim 3 (depends on 1)

The computing system of claim 1, wherein the processor is further configured to, at development time:
receive a plurality of processing device type indicators for the plurality of layers of the machine learning model topology; and
select the plurality of processing devices such that the plurality of processing devices match the plurality of processing device type indicators.

### Claim 4 (depends on 3)

The computing system of claim 3, wherein the processor is further configured to:
receive a respective plurality of performance targets for the plurality of layers of the machine learning model topology;
determine that no available configuration of processing devices communicatively coupled to the computing system both matches the plurality of processing device type indicators and meets the plurality of performance targets; and
generate:
a performance-modified inferencer graph indicating a performance-modified processing device configuration that matches the plurality of processing device type indicators and does not meet the plurality of performance targets; or
a device-modified inferencer graph indicating a device-modified processing device configuration that meets the plurality of performance targets and does not match the plurality of processing device type indicators.

### Claim 5 (depends on 1)

The computing system of claim 1, wherein generating the inferencer graph includes, for a first internal representation layer and a second internal representation layer of the internal representation graph that both output to a third internal representation layer, setting the output buffer sizes of the corresponding inferencer layer blocks to be equal to each other.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein generating the inferencer graph includes applying bias channel reduction, 3×3 Winograd convolution, 5×5 complex-domain Winograd convolution, or filter-sparsity packing and runtime skipping to at least one internal representation layer of the plurality of internal representation layers.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein generating the inferencer graph includes selecting an input ordering for a plurality of inferencer layer inputs to at least one inferencer layer block of the plurality of inferencer layer blocks.

### Claim 8 (depends on 1)

The computing system of claim 1, wherein generating the inferencer graph includes determining a respective memory layout of memory allocated to each inferencer layer block of the plurality of inferencer layer blocks.

### Claim 9 (depends on 1)

The computing system of claim 1, wherein generating the inferencer graph includes allocating at least one digital signal processor (DSP) to perform the logic function of at least one inferencer layer block of the plurality of inferencer layer blocks.

### Claim 10 (depends on 1)

The computing system of claim 1, wherein the plurality of layers included in the machine learning model topology have two or more respective computation precisions.

### Claim 11 (depends on 1)

The computing system of claim 1, wherein:
generating the inferencer graph includes dividing the internal representation graph into at least a first internal representation subgraph and a second internal representation subgraph; and
at deployment time, the processor is configured to transmit instructions to implement the first internal representation subgraph to a first subset of the plurality of processing devices and transmit instructions to implement the second internal representation subgraph to a second subset of the plurality of processing devices.

### Claim 12 (depends on 1)

The computing system of claim 1, wherein the processor and the plurality of processing devices are located in a data center and connected by wired connections.

### Claim 13 (depends on 1)

The computing system of claim 1, wherein each processing device of the plurality of processing devices is a central processing unit (CPU), a graphical processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

### Claim 14

A method for use with a computing system, the method comprising:
at development time:
receiving a machine learning model topology including a plurality of layers; and
generating an internal representation graph of the machine learning model topology, wherein the internal representation graph includes a plurality of internal representation layers; and
at least in part by performing one or more modifications to the internal representation graph, generating an inferencer graph including a plurality of inferencer layer blocks, wherein each inferencer layer block indicates an input buffer size, a logic function, and an output buffer size; and
at deployment time, transmitting, to a plurality of processing devices, instructions to implement the machine learning model topology with the respective input buffer sizes, logic functions, and output buffer sizes indicated in the plurality of inferencer layer blocks of the inferencer graph.

### Claim 15 (depends on 14)

The method of claim 14, further comprising, at development time:
receiving a respective plurality of performance targets for the plurality of layers of the machine learning model topology; and
generating the inferencer graph such that the plurality of inferencer layer blocks meet the respective plurality of performance targets.

### Claim 16 (depends on 14)

The method of claim 14, further comprising, at development time:
receiving a plurality of processing device type indicators for the plurality of layers of the machine learning model topology; and
selecting the plurality of processing devices such that the plurality of processing devices match the plurality of processing device type indicators.

### Claim 17 (depends on 14)

The method of claim 14, wherein generating the inferencer graph includes, for a first internal representation layer and a second internal representation layer of the internal representation graph that both output to a third internal representation layer, setting the output buffer sizes of the corresponding inferencer layer blocks to be equal to each other.

### Claim 18 (depends on 14)

The method of claim 14, wherein generating the inferencer graph includes selecting an input ordering for a plurality of inferencer layer inputs to at least one inferencer layer block of the plurality of inferencer layer blocks.

### Claim 19 (depends on 14)

The method of claim 14, wherein generating the inferencer graph includes determining a respective memory layout of memory allocated to each inferencer layer block of the plurality of inferencer layer blocks.

### Claim 20

A computing system comprising:
a processor configured to:
at development time:
receive a machine learning model topology including a plurality of layers;
receive a respective plurality of performance targets for the plurality of layers of the machine learning model topology;
receive a plurality of processing device type indicators for the plurality of layers of the machine learning model topology; and
based at least in part on the machine learning model topology, the plurality of performance targets, and the plurality of processing device indicators, generate an inferencer graph including a plurality of inferencer layer blocks, wherein each inferencer layer block indicates an input buffer size, a logic function, and an output buffer size; and
at deployment time, transmit, to a plurality of processing devices that match the plurality of processing device type indicators, instructions to implement the machine learning model topology with the respective input buffer sizes, logic functions, and output buffer sizes indicated in the plurality of inferencer layer blocks of the inferencer graph, wherein each inferencer layer block of the inferencer graph meets the performance target for the corresponding layer of the machine learning model topology.