# US 2021/0264322 A1 · Computer-implemented systems, methods and computer programs for adapting a machine-learning-architecture and for processing input data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264322 A1
- Document ID: US20210264322A1
- Status: Published application
- Kind: A1
- Application number: 17/179,451
- Filed: 2021-02-19
- Published: 2021-08-26
- Earliest claimed: 2020-02-26
- Assignees: Sony Semiconductor Solutions Corporation
- Applicants: Sony Semiconductor Solutions Corporation
- Inventors: Lev MARKHASIN; Shangyin GAO
- Web page: https://patentgrep.com/patent/US20210264322A1

## CPC

- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06F 18/211: Selection of the most significant subset of features
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/25: Fusion techniques
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 20/20: Ensemble learning
- G06V 10/70: using pattern recognition or machine learning (optical pattern recognition or electronic computations therefor G06V10/88)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/80: Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level (multimodal speaker identification or verification G10L17/10)
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Examples relate to a computer-implemented system, a computer-implemented method and a computer program for adapting a machine-learning-architecture, and to a computer-implemented system, a computer-implemented method and a computer program and for processing input data using a machine-learning model having a machine-learning architecture. The computer-implemented system for adapting a machine-learning architecture of a first machine-learning model comprises one or more processors and one or more storage devices. The machine-learning architecture comprises a color image branch configured to process color image data. The color image branch comprises a sequence of convolution blocks. The machine-learning architecture comprises a depth branch configured to process depth data. The depth branch comprises a sequence of convolution blocks. The machine-learning architecture comprises one or more fusing components configured to combine intermediary data of two or more of the branches. Each fusing component is configured to combine an output of a first block of one of the branches and an output of a second block of another branch. An output of the fusing component is used as input of a third block of one of the branches. The third block is a convolution block of one of the sequences of convolution blocks. The machine-learning architecture comprises an output component configured to provide an output of the machine-learning model. The output is based on an output of one or more of the branches. The system is configured to adapt the machine-learning architecture of the first machine-learning model using a second machine-learning model. The second machine-learning model is trained to select, for each of the one or more fusing components, the first block, the second block, and the third block.

## Claims

### Claim 1

A computer-implemented system for adapting a machine-learning architecture of a first machine-learning model, the system comprising one or more processors and one or more storage devices, the machine-learning architecture comprising:
a color image branch configured to process color image data, the color image branch comprising a sequence of convolution blocks;
a depth branch configured to process depth data, the depth branch comprising a sequence of convolution blocks;
one or more fusing components configured to combine intermediary data of two or more of the branches, each fusing component being configured to combine an output of a first block of one of the branches and an output of a second block of another branch, an output of the fusing component being used as input of a third block of one of the branches, the third block being a convolution block of one of the sequences of convolution blocks; and
an output component, configured to provide an output of the machine-learning model, the output being based on an output of one or more of the branches,
wherein the system is configured to adapt the machine-learning architecture of the first machine-learning model using a second machine-learning model, wherein the second machine-learning model is trained to select, for each of the one or more fusing components, the first block, the second block, and the third block.

### Claim 2 (depends on 1)

The system according to claim 1, wherein the second machine-learning model is further trained to select, for each of the one or more fusing components, a combination operator being used by the fusing component.

### Claim 3 (depends on 1)

The system according to claim 1, wherein the second machine-learning model is further trained to select a number of fusing components to use in the machine-learning architecture.

### Claim 4 (depends on 1)

The system according to claim 1, wherein the machine-learning architecture comprises a combined branch configured to process combined color image and depth data, the combined branch comprising a sequence of convolution blocks, an input to the combined branch being provided by one of the fusing components.

### Claim 5 (depends on 4)

The system according to claim 4, wherein the one fusing component is configured to generate the combined color image and depth data based on an output of a convolution block of the color image branch and based on an output of a convolution block of the depth branch, wherein the one fusing component is configured to generate the combined color image and depth data using an attention mechanism that is based on the output of the convolution block of the color image branch and based on the output of the convolution block of the depth branch.

### Claim 6 (depends on 4)

The system according to claim 4, wherein the one or more fusing components comprise a subset of fusing components being configured to combine an output of a first block of one of the branches, an output of a second block of another branch, and an output of a fourth block of a further branch, an output of the fusing component being used as input of a third block of one of the branches, wherein the machine-learning model is further trained to select, for each fusing component of the subset, the fourth block.

### Claim 7 (depends on 1)

The system according to claim 1, wherein the second machine-learning model is further trained to select a number of convolution blocks of the sequences of convolution blocks.

### Claim 8 (depends on 1)

The system according to claim 1, wherein the system is configured to iteratively adapt the machine-learning architecture by determining a performance estimate of the first machine-learning model being based on the machine-learning architecture, providing the performance estimate as input to the second machine-learning model, using an output of the second machine-learning model to adapt the machine-learning architecture, determining a performance estimate of the machine-learning model being based on the adapted machine-learning architecture, and repeating the process until a termination condition is met.

### Claim 9 (depends on 8)

The system according to claim 8, wherein the performance estimate of the machine-learning model is determined using a set of training data, the set of training data comprising color image data, depth data, and desired output data, wherein the system is configured to trigger the adaption of the machine-learning architecture if the set of training data is changed, and wherein the system is configured to generate the first machine-learning model based on the adapted machine-learning architecture after the adaption of the machine-learning architecture, and to train the first machine-learning model based on the set of training data.

### Claim 10 (depends on 9)

The system according to claim 9, wherein the system is configured to process input data comprising color image data and depth data using the first machine-learning model.

### Claim 11

A computer-implemented system for processing color image data and depth data, the system comprising one or more processors and one or more storage devices, wherein the system is configured to process the color image data and the depth image data using a machine-learning model having a machine-learning architecture, the machine-learning architecture comprising:
a color image branch configured to process the color image data, the color image branch comprising a sequence of convolution blocks;
a depth branch configured to process the depth data, the depth branch comprising a sequence of convolution blocks;
a combined branch configured to process combined color image and depth data, the combined branch comprising a sequence of convolution blocks;
one or more fusing components configured to generate the combined color image and depth data based on an output of a convolution block of the color image branch and based on an output of a convolution block of the depth branch, wherein the one or more fusing components are each configured to generate the combined color image and depth data using an attention mechanism that is based on the output of the convolution block of the color image branch and based on the output of the convolution block of the depth branch; and
an output component, configured to provide an output of the machine-learning model, the output being based at least on an output of the combined branch.

### Claim 12 (depends on 11)

The system according to claim 11, wherein the output component is configured to provide the output further based on an output of the color image branch and/or based on an output of the depth branch.

### Claim 13 (depends on 12)

The system according to claim 12, wherein the output component is configured to perform a combination operation based on the output of the combined branch and based on the output of at least one of the color image branch and the depth branch.

### Claim 14 (depends on 13)

The system according to claim 13, wherein the combination operation is an averaging operation.

### Claim 15 (depends on 11)

The system according to claim 11, wherein at least the combined branch comprises one or more detection head layers and a prediction layer, the prediction layer being configured to provide an output of the respective branch based on an output of the one or more detection head layers, the one or more detection head layers being configured to process an output of the one or more convolution blocks of the respective branch.

### Claim 16

A computer-implemented method for adapting a machine-learning architecture of a first machine-learning model, the machine-learning architecture comprising:
a color image branch configured to process color image data, the color image branch comprising a sequence of convolution blocks,
a depth branch configured to process depth data, the depth branch comprising a sequence of convolution blocks,
one or more fusing components, configured to combine intermediary data of two or more of the branches, each fusing component being configured to combine an output of a first block of one of the branches and an output of a second block of another branch, an output of the fusing component being used as input of a third block of one of the branches, the third block being a convolution block of one of the sequences of convolution blocks, and
an output component, configured to provide an output of the machine-learning model, the output being based on an output of one or more of the branches,
wherein the method comprises adapting the machine-learning architecture of the first machine-learning model using a second machine-learning model, wherein the second machine-learning model is trained to select, for each of the one or more fusing components, the first block, the second block, and the third block.

### Claim 17

A computer-implemented method for processing color image data and depth data, the method comprising processing the color image data and the depth image data using a machine-learning model having a machine-learning architecture, the machine-learning architecture comprising:
a color image branch configured to process the color image data, the color image branch comprising a sequence of convolution blocks;
a depth branch configured to process the depth data, the depth branch comprising a sequence of convolution blocks;
a combined branch configured to process combined color image and depth data, the combined branch comprising a sequence of convolution blocks;
one or more fusing components configured to generate the combined color image and depth data based on an output of a convolution block of the color image branch and based on an output of a convolution block of the depth branch, wherein the one or more fusing components are each configured to generate the combined color image and depth data using an attention mechanism that is based on the output of the convolution block of the color image branch and based on the output of the convolution block of the depth branch; and
an output component, configured to provide an output of the machine-learning model, the output being based at least on an output of the combined branch.

### Claim 18 (depends on 16)

A computer program having a program code for performing the method of claim 16, when the computer program is executed on a computer, a processor, or a programmable hardware component.

### Claim 19 (depends on 17)

A computer program having a program code for performing the method of claim 17, when the computer program is executed on a computer, a processor, or a programmable hardware component.
