# US 2023/0274151 A1 · Multi-modal neural network architecture search

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274151 A1
- Document ID: US20230274151A1
- Status: Published application
- Kind: A1
- Application number: 17/915,796
- Filed: 2021-03-30
- Published: 2023-08-31
- Earliest claimed: 2020-03-30
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Zhen Xu; David Richard So; Andrew M. Dai
- Web page: https://patentgrep.com/patent/US20230274151A1

## CPC

- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/086: using evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/09: Supervised learning
- G06N 3/092: Reinforcement learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for searching for an architecture for a neural network that performs a multi-modal task that requires operating on inputs that each include data from multiple different modalities.

## Claims

### Claim 1

A method comprising:
receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples, wherein each training example comprises a respective network input from each of a plurality of modalities; and
determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
generating a plurality of candidate neural network architectures, wherein:
each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state, and
generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture:
selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;
determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task; and
selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitnesses.

### Claim 2

The method of any preceding claim, wherein the plurality of operation blocks are ordered in a sequence, and wherein, for each of the operation blocks, the respective set of possible inputs includes each of the respective network inputs from each of the plurality of modalities and the block outputs of any of the operation blocks that are before the operation block in the sequence.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
each candidate neural network architecture (i) generates an output state that is a combination of a respective output state for each of the plurality of modalities and (ii) processes at least the output state using one or more neural network layers to generate a final output for the multi-modal machine learning task, and
the respective output state for each of the plurality of modalities is generated from output hidden states that are both (i) not used in a block input for any of the plurality of operation blocks and (ii) conditioned on the respective network input from the modality.

### Claim 4 (depends on 3)

The method of claim 3, wherein the output state is a concatenation of the respective output states for each of the modalities.

### Claim 5 (depends on 1)

The method of claim 1 wherein determining the fitness comprises:
training a new neural network having the candidate neural network architecture on a training subset of the training data; and
determining the fitness by evaluating a performance of the trained new neural network on a validation subset of the training data.

### Claim 6 (depends on 1)

The method of claim 1, wherein each training example is data from an electronic health record and wherein the plurality of modalities comprise two or more of: contextual features, categorical features, continuous features, or clinical notes.

### Claim 7 (depends on 1)

The method of claim 1, wherein each training example is health data for a patient, and wherein the plurality of modalities includes one or more of: data of one or more modalities extracted from an electronic health record for the patient, medical image data for the patient, genomics data for the patient, or waveforms of speech or other audio data relevant to the patient.

### Claim 8 (depends on 6)

The method of claim 6, wherein the multi-modal machine learning task is a task to predict an aspect of health of a patient from electronic health record data or health data for the patient.

### Claim 9 (depends on 1)

The method of claim 1, wherein the plurality of modalities include audio data and corresponding video data and the multi-modal machine learning task is audio-video speech recognition or visual question-answering.

### Claim 10 (depends on 1)

The method of claim 1, wherein the plurality of modalities include content data and one or more of meta data for the content data or history data associated with a user being presented the content data, and wherein the multi-modal machine learning task is content recommendation.

### Claim 11 (depends on 1)

The method of claim 1, wherein generating the plurality of candidate neural network architectures comprises:
maintaining population data comprising, for each candidate neural network architecture in a population of candidate neural network architectures, data defining the candidate neural network architecture and the fitness for the candidate neural network architecture, and
repeatedly performing the following operations:
selecting a candidate neural network architecture from the population,
generating a new candidate neural network architecture by applying one or more mutations to the selected candidate neural network architecture having the best fitness,
determining a fitness for the new candidate neural network architecture, and
adding the new candidate neural network architecture to the population.

### Claim 12 (depends on 11)

The method of claim 11, the following operations further comprising:
selecting another plurality of candidate neural network architectures from the population, and
removing from the population the candidate neural network architecture from the selected other plurality of candidate neural network architectures having a fitness that was determined least recently.

### Claim 13 (depends on 11)

The method of claim 11, wherein selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitnesses comprises, after repeatedly performing the operations: selecting the optimized neural network architecture from the population based on the fitnesses of the candidate neural network architectures in the population.

### Claim 14 (depends on 13)

The method of claim 13, wherein the selecting the optimized neural network architecture comprises selecting the candidate neural network architecture in the population that has the best fitness.

### Claim 15 (depends on 13)

The method of claim 13, wherein the selecting comprises:
selecting, from the population, a plurality of candidate neural network architectures having the highest fitnesses;
training respective neural networks having each of the plurality of selected candidate neural network architectures;
determining a respective fitness for each of the trained respective neural networks; and
selecting, as the optimized neural network architecture, the candidate neural network architecture corresponding to the trained respective neural network having the best fitness.

### Claim 16 (depends on 1)

The method of claim 1, wherein determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task comprises:
training a neural network having the candidate neural network architecture on at least a portion of the training data; and
determining a fitness for the trained neural network based on a fitness measure for the multi-modal machine learning task.

### Claim 17 (depends on 1)

The method of claim 1, further comprising
providing data specifying the optimized architecture for use in performing the multi-modal machine learning task.

### Claim 18 (depends on 1)

The method of claim 1, further comprising:
using a neural network having the optimized architecture to perform the multi-modal machine learning task on new examples.

### Claim 19 (depends on 1)

The method of claim 1, wherein generating each of the plurality of candidate neural network architectures further comprises, for each of the operation blocks in the candidate neural network architecture:
selecting respective values for each of one or more search fields that define the one or more operations that are applied by the operation block to the block input to the operation block.

### Claim 20

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples, wherein each training example comprises a respective network input from each of a plurality of modalities; and
determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
generating a plurality of candidate neural network architectures, wherein:
each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state, and
generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture:
selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;
determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task; and
selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitness.

### Claim 21

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:
receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples, wherein each training example comprises a respective network input from each of a plurality of modalities; and
determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
generating a plurality of candidate neural network architectures, wherein:
each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state, and
generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture:
selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;
determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task; and
selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitness