# US 11,908,193 B2 · Obtaining custom artificial neural network architectures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,193 B2
- Document ID: US11908193B2
- Status: Granted
- Kind: B2
- Application number: 17/097,249
- Filed: 2020-11-13
- Published: 2021-10-21
- Granted: 2024-02-20
- Earliest claimed: 2020-04-17
- Assignees: Blaize, Inc.
- Applicants: Blaize, Inc.
- Inventors: Ilya A. Balabin; Adam P. Geringer; Dmitry Zakharchenko
- Web page: https://patentgrep.com/patent/US11908193B2

## CPC

- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/046: Forward inferencing; Production systems
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/1365: {Matching; Classification}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems and apparatuses for a custom artificial neural network (ANN) architecture are disclosed. One method includes selecting existing ANN architectures, calculating ANN architecture fingerprints, calculating ANN architecture pairwise similarities among the existing ANN architectures, calculating centrality scores for the existing ANN architectures using the ANN architecture pairwise similarities, calculating dataset pairwise similarities between the target dataset and each of the existing datasets using dataset fingerprints, calculating target performance scores for the existing ANN architectures on the target dataset using performance scores of the existing ANN architectures on the existing datasets and the dataset pairwise similarities, calculating interpolation weights for the existing ANN architectures using the target performance scores of the existing ANN architectures on the target dataset and the centrality scores, and obtaining the custom ANN architecture by interpolating among the existing ANN architectures using the calculated interpolation weights.

## Claims

### Claim 1

A method for obtaining a custom artificial neural network (ANN) architecture for performing an inference task on a target dataset of a video data stream with specified performance criteria, comprising:
selecting a plurality of existing ANN architectures of a same architecture type and a plurality of existing datasets of a same data type as the target dataset of the video stream of data;
calculating a plurality of ANN architecture fingerprints including an ANN architecture fingerprint of each of the plurality of the existing ANN architectures;
calculating ANN architecture pairwise similarities among the plurality of existing ANN architectures using the plurality of ANN architecture fingerprints;
calculating centrality scores for the plurality of existing ANN architectures using the ANN architecture pairwise similarities;
calculating dataset fingerprints including a dataset fingerprint of the target dataset and each of the plurality of existing datasets;
calculating dataset pairwise similarities between the target dataset and each of the plurality of existing datasets using the dataset fingerprints;
obtaining performance scores of the existing ANN architectures on the existing datasets;
calculating target performance scores for the plurality of existing ANN architectures on the target dataset using the performance scores of the plurality of existing ANN architectures on the existing datasets and the dataset pairwise similarities;
calculating interpolation weights for the plurality of existing ANN architectures using the target performance scores of the existing ANN architectures on the target dataset and the centrality scores calculated using the ANN pairwise similarities;
obtaining the custom ANN architecture by interpolating among the plurality of existing ANN architectures using the calculated interpolation weights;
sensing, by one or more cameras, the stream of video data;
detecting, categorizing, and tracking, by the custom ANN architecture, objects within the stream of video data; and
controlling an autonomous driving car based at least on the detected, categorized, and tracked objects.

### Claim 2 (depends on 1)

The method of claim 1, wherein calculating interpolation weights for the plurality of existing ANN architectures using the target performance scores of the existing ANN architectures on the target dataset and the centrality scores calculated using the ANN pairwise similarities comprises calculating products of an estimated performance score for each of the plurality of existing ANN architectures on the target dataset and a centrality score for a same ANN architecture calculated using the pairwise similarities among the plurality of existing ANN architectures.

### Claim 3 (depends on 2)

The method of claim 2, wherein the centrality scores includes at least one of an ANN closeness centrality, a degree centrality, or a harmonic centrality.

### Claim 4 (depends on 2)

The method of claim 2, wherein calculating target performance scores for the plurality of existing ANN architectures on the target dataset using the performance scores of the plurality of existing ANN architectures on the known datasets and the dataset pairwise similarities comprises interpolating among the performance scores of the plurality of existing ANN architectures on the existing datasets using the dataset pairwise similarities between the target dataset and each of the existing datasets as weights.

### Claim 5 (depends on 4)

The method of claim 4, wherein calculating the dataset pairwise similarity between any two datasets of the target data set and the existing data set comprises calculating convolutions of two dataset fingerprints of the any two datasets with a similarity metric.

### Claim 6 (depends on 5)

The method of claim 5, wherein calculating the dataset fingerprints comprises calculating a fixed-length real number vectors by mapping a dataset of the target dataset or any of the existing datasets onto a latent space of an existing ANN architecture pre-trained on one or more other datasets of a same type.

### Claim 7 (depends on 1)

The method of claim 1, wherein calculating ANN architecture pairwise similarities among the plurality of existing ANN architectures using the ANN architecture fingerprints comprises calculating convolutions of any two of the plurality of ANN architecture fingerprints with a similarity metric.

### Claim 8 (depends on 7)

The method of claim 7, wherein calculating a plurality of ANN architecture fingerprints including an ANN architecture fingerprint of each of the plurality of the existing ANN architectures comprises obtaining data structures that contains a complete description of an ANN topology and hyperparameters, wherein the ANN topology comprises a map of an ANN layer structure and pairwise connections among nodes of the ANN layer structure.

### Claim 9 (depends on 1)

The method of claim 1, further including determining that the plurality of existing ANN architectures are the same type comprising determining that the plurality of existing ANN architectures use as an input datasets of same type and perform inference tasks of a same type, and further including determining that the plurality of existing data types are a same type as the target dataset comprising determining objects within the existing dataset types and the target dataset are of a same type and that mathematical descriptions of data points of the existing dataset types and the target dataset have a same format.

### Claim 10 (depends on 1)

The method of claim 1, wherein the obtained performance scores of the existing ANN architectures provide a characterization of at least one of an accuracy, a computing speed, a memory size, or other characteristics of an inference calculation.

### Claim 11

A system of an autonomous car for obtaining a custom artificial neural network (ANN) architecture for performing an inference task on a target dataset of a video data stream with specified performance criteria, comprising:
one or more cameras configured to sense a stream of video data;
a database;
one or more computing devices connected through a network to the database,
memory including instructions that, when executed by the one or more computing devices, enables the system of the autonomous car to:
select a plurality of existing ANN architectures of a same architecture type and a plurality of existing datasets of a same data type as the target dataset of the video data stream from the database;
calculate a plurality of ANN architecture fingerprints including an ANN architecture fingerprint of each of the plurality of the existing ANN architectures;
calculate ANN architecture pairwise similarities among the plurality of existing ANN architectures using the plurality of ANN architecture fingerprints;
calculate centrality scores for the plurality of existing ANN architectures using the ANN architecture pairwise similarities;
calculate dataset fingerprints including a dataset fingerprint of the target dataset and each of the plurality of existing datasets;
calculate dataset pairwise similarities between the target dataset and each of the plurality of existing datasets using the dataset fingerprints;
obtain performance scores of the existing ANN architectures on the existing datasets from the database;
calculate target performance scores for the plurality of existing ANN architectures on the target dataset using the performance scores of the plurality of existing ANN architectures on the existing datasets and the dataset pairwise similarities;
calculate interpolation weights for the plurality of existing ANN architectures using the target performance scores of the existing ANN architectures on the target dataset and the centrality scores calculated using the ANN pairwise similarities; and
obtain the custom ANN architecture by interpolating among the plurality of existing ANN architectures using the calculated interpolation weights;
wherein the custom ANN architecture is configured to detect, categorize, and track, objects within the stream of video data; and
wherein the instructions when executed by the one or more computing devices further enables the system of the autonomous car to control the autonomous driving car based at least on the detected, categorized, and tracked objects.

### Claim 12 (depends on 11)

The system of claim 11, wherein calculating interpolation weights for the plurality of existing ANN architectures using the target performance scores of the existing ANN architectures on the target dataset and the centrality scores calculated using the ANN pairwise similarities comprises calculating products of an estimated performance score for each of the plurality of existing ANN architectures on the target dataset and a centrality score for the same ANN architecture calculated using the pairwise similarities among the plurality of existing ANN architectures.

### Claim 13 (depends on 12)

The system of claim 12, wherein the centrality scores includes at least one of an ANN closeness centrality, a degree centrality, or a harmonic centrality.

### Claim 14 (depends on 12)

The system of claim 12, wherein calculating target performance scores for the plurality of existing ANN architectures on the target dataset using the performance scores of the plurality of existing ANN architectures on the known datasets and the dataset pairwise similarities comprises interpolating among the performance scores of the plurality of existing ANN architectures on the existing datasets using the dataset pairwise similarities between the target dataset and each of the existing datasets as weights.

### Claim 15 (depends on 14)

The system of claim 14, wherein calculating the dataset pairwise similarity between any two datasets of the target data set and the existing data set comprises calculating convolutions of two dataset fingerprints of the any two datasets with a similarity metric.

### Claim 16 (depends on 15)

The system of claim 15, wherein calculating the dataset fingerprints comprises calculating a fixed-length real number vectors by mapping a dataset of the target dataset or anyone of the existing data sets onto a latent space of an existing ANN architecture pre-trained on one or more other datasets of a same type.

### Claim 17 (depends on 11)

The system of claim 11, wherein calculating ANN architecture pairwise similarities among the plurality of existing ANN architectures using the ANN architecture fingerprints comprises calculating convolutions of any two of the plurality of ANN architecture fingerprints with a similarity metric.

### Claim 18 (depends on 17)

The system of claim 17, wherein calculating a plurality of ANN architecture fingerprints including an ANN architecture fingerprint of each of the plurality of the existing ANN architectures comprises obtaining data structures that contains a complete description of an ANN topology, and hyperparameters, wherein the ANN topology comprises a map of an ANN layer structure and pairwise connections among the ANN nodes.

### Claim 19 (depends on 11)

The system of claim 11, wherein execution by the one or more computing devices, further enables the system to:
determine that the plurality of existing ANN further including determining that the plurality of existing ANN architectures are the same type comprising determining that the plurality of existing ANN architectures use as an input datasets of same type and perform inference tasks of a same type; and
determine that the plurality of existing data types is a same type as the target dataset comprising determining objects within the existing dataset types and the target dataset are of a same type and that mathematical descriptions of data points of the existing dataset types and the target dataset have a same format.

### Claim 20 (depends on 11)

The system of claim 11, wherein the obtained performance scores of the existing ANN architectures provide a characterization of at least one of an accuracy, a computing speed, a memory size, or other characteristics of an inference calculation.