# US 2024/0256892 A1 · Method and system for federated learning training for a neural network associated with autonomous vehicles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256892 A1
- Document ID: US20240256892A1
- Status: Published application
- Kind: A1
- Application number: 18/159,767
- Filed: 2023-01-26
- Published: 2024-08-01
- Earliest claimed: 2023-01-26
- Assignees: Woven By Toyota, Inc.
- Applicants: Woven By Toyota, Inc.
- Inventors: Yuki KAWANA; Yusuke YACHIDE; Takaaki TAGAWA; Koichiro YAMAGUCHI; Daisuke HASHIMOTO; Hiroyuki AONO; Ryo TAKAHASHI
- Web page: https://patentgrep.com/patent/US20240256892A1

## CPC

- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method includes receiving a first model and collecting sensor data acquired by a sensor on a first vehicle. The method also includes identifying a first data item from among the collected sensor data when the first data item is determined to satisfy a criterion. The method further include detecting an object contained in the identified first data item by running the first model with the identified first data item as input and establishing communication with a computer on a second vehicle located at equal to or less than a predetermined distance from the first vehicle. The method also includes receiving a second data item that is indicated as containing the object from the computer on the second vehicle and generating a training dataset. The method further includes training with respect to the first model on the training dataset and transmitting first data representing the trained first model.

## Claims

### Claim 1

A method, implemented by programmed one or more processors, comprising:
receiving, from one or more server computers through a communication network, a first model;
collecting sensor data acquired by a sensor on a first vehicle;
identifying a first data item from among the collected sensor data when the first data item is determined to satisfy a criterion;
detecting an object contained in the identified first data item by running the first model with the identified first data item as input to the first model;
establishing communication with a computer on a second vehicle located at equal to or less than a predetermined distance from the first vehicle;
receiving a second data item that is indicated as containing the object from the computer on the second vehicle;
generating a training dataset containing the first data item, the second data item and a label of the object as a supervision signal;
training with respect to the first model on the training dataset; and
transmitting first data representing the trained first model to the one or more server computers though the communication network.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising:
receiving, from the one or more server computers through a communication network, update data that represents a model that is trained with aggregated model information from other edge models; and
updating the first model based on the update data.

### Claim 3 (depends on 1)

The method according to claim 1, wherein the training with respect to the first model comprises training a copy of the received first model.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising obtaining, as the first data, a gradient between the first model prior to the training and the first model subsequent to the training.

### Claim 5 (depends on 3)

The method according to claim 3, further comprising obtaining, as the first data, a gradient between the received first model and the copy of the first model that is updated by the training.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the receiving the second data item comprises receiving the second data item and an inference result of a second model in the second vehicle with respect to detecting of the object in the second data item.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the generating the training dataset comprises obtaining the label of the object by combining inference results of the first model and a second model in the second vehicle that detects the object in the second data item.

### Claim 8

A computing device, comprising:
a memory storing instructions; and
a processor configured to execute the instructions to:
receive, from one or more server computers through a communication network, a first model;
collect sensor data acquired by a sensor on a first vehicle;
identify a first data item from among the collected sensor data when the first data item is determined to satisfy a criterion;
detect an object contained in the identified first data item by running the first model with the identified first data item as input to the first model;
establish communication with a computer on a second vehicle located at equal to or less than a predetermined distance from the first vehicle;
receive a second data item that is indicated as containing the object from the computer on the second vehicle;
generate a training dataset containing the first data item, the second data item and a label of the object as a supervision signal;
train with respect to the first model on the training dataset; and
transmit first data representing the trained first model to the one or more server computers though the communication network.

### Claim 9 (depends on 8)

The computing device according to claim 8, wherein the processor is further configured to execute the instructions to:
receive, from the one or more server computers through a communication network, update data that represents a model that is trained with aggregated model information from other edge models; and
update the first model based on the update data.

### Claim 10 (depends on 8)

The computing device according to claim 8, wherein the instructions to train with respect to the first model comprises instructions to train a copy of the received first model.

### Claim 11 (depends on 8)

The computing device according to claim 8, wherein the processor is further configured to execute the instructions to obtain, as the first data, a gradient between the first model prior to the training and the first model subsequent to the training.

### Claim 12 (depends on 10)

The computing device according to claim 10, wherein the processor is further configured to execute the instructions to obtain, as the first data, a gradient between the received first model and the copy of the first model that is updated by the training.

### Claim 13 (depends on 8)

The computing device according to claim 8, wherein the instructions to receive the second data item comprises instructions to receive the second data item and an inference result of a second model in the second vehicle with respect to detecting of the object in the second data item.

### Claim 14 (depends on 8)

The computing device according to claim 8, wherein the instructions to generate the training dataset comprises instructions to obtain the label of the object by combining inference results of the first model and a second model in the second vehicle that detects the object in the second data item.

### Claim 15

A non-transitory computer-readable medium storing instructions, the instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to:
receive, from one or more server computers through a communication network, a first model;
collect sensor data acquired by a sensor on a first vehicle;
identify a first data item from among the collected sensor data when the first data item is determined to satisfy a criterion;
detect an object contained in the identified first data item by running the first model with the identified first data item as input to the first model;
establish communication with a computer on a second vehicle located at equal to or less than a predetermined distance from the first vehicle;
receive a second data item that is indicated as containing the object from the computer on the second vehicle;
generate a training dataset containing the first data item, the second data item and a label of the object as a supervision signal;
train with respect to the first model on the training dataset; and
transmit first data representing the trained first model to the one or more server computers though the communication network.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise: one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to:
receive, from the one or more server computers through a communication network, update data that represents a model that is trained with aggregated model information from other edge models; and
update the first model based on the update data.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein causing the one or more processors to train with respect to the first model comprises causing the one or more processors to train a copy of the received first model.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise: one or more instructions that, when executed by one or more processors of a device, cause the one or more processors to obtain, as the first data, a gradient between the first model prior to the training and the first model subsequent to the training.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein causing the one or more processors to receive the second data item comprises causing the one or more processors to receive the second data item and an inference result of a second model in the second vehicle with respect to detecting of the object in the second data item.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein causing the one or more processors to generate the training dataset comprises causing the one or more processors to obtain the label of the object by combining inference results of the first model and a second model in the second vehicle that detects the object in the second data item.
