# US 11,308,363 B2 · Device and method for training an object detection model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,363 B2
- Document ID: US11308363B2
- Status: Granted
- Kind: B2
- Application number: 16/830,447
- Filed: 2020-03-26
- Published: 2020-07-16
- Granted: 2022-04-19
- Earliest claimed: 2020-03-26
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Julio Jarquin Arroyo; Kay-Ulrich Scholl
- Web page: https://patentgrep.com/patent/US11308363B2

## CPC

- G06N 3/08: Learning methods
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06V 10/7747: {Organisation of the process, e.g. bagging or boosting}
- G06V 10/776: Validation; Performance evaluation
- G06V 10/806: {of extracted features}
- G06V 2201/07: Target detection
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A training device may include one or more processors configured to generate, using a data augmentation model, augmented sensor data for sensor data, the sensor data provided by a plurality of sensors, wherein the augmented sensor data comprise error states of one or more sensors of the plurality of sensors providing the sensor data, and to train an object detection model based on the augmented sensor data.

## Claims

### Claim 1

A training device, comprising:
one or more processors, configured to:
generate, using a data augmentation model, augmented sensor data for sensor data, the sensor data provided by a plurality of sensors, wherein the augmented sensor data comprise error states of one or more sensors of the plurality of sensors providing the sensor data; and
train an object detection model based on the augmented sensor data, wherein the object detection model comprises a plurality of feature extraction models, and wherein the one or more processors are configured to:
determine, based on the plurality of feature extraction models, a plurality of features for the augmented sensor data; and
train the object detection model based on the plurality of features, wherein the one or more processors are configured to train the object detection model by suppressing one or more selected features of the plurality of features, wherein the one or more selected features comprises all features associated with one or more selected sensors.

### Claim 2 (depends on 1)

The training device of claim 1, wherein the error states comprise one or more of: weather conditions, an occlusion, sensor data noise, a sensor saturation, a sensor bias, a drift, and/or a scaling.

### Claim 3 (depends on 1)

The training device of claim 1, wherein the one or more processors are configured to randomly select the one or more selected features of the plurality of features.

### Claim 4 (depends on 1)

The training device of claim 1, wherein the sensor data comprise information about at least one object, wherein each sensor data item of the plurality of sensor data items comprises a plurality of elements, and wherein one or more elements of the plurality of elements are associated with the at least one object.

### Claim 5 (depends on 1)

The training device of claim 1, wherein the object detection model comprises a neural network.

### Claim 6 (depends on 1)

The training device of claim 1, wherein the one or more processors are further configured to:
determine, based on the plurality of feature extraction models, a plurality of feature vectors for the sensor data provided by the plurality of sensors;
fuse the plurality of feature vectors except for the feature vectors associated with the one or more selected sensors to a fused feature vector; and
train the object detection model using the fused feature vector.

### Claim 7 (depends on 1)

The training device of claim 1, wherein the plurality of feature extraction models includes a convolutional feature extraction, a spatial point feature extraction, a feature encoder, or any combination of these.

### Claim 8 (depends on 4)

The training device of claim 4, wherein the object detection model comprises a plurality of feature extraction models, and wherein the one or more processors are configured to determine, based on the plurality of feature extraction models, a plurality of features for the augmented sensor data, and to train the object detection model based on the plurality of features, wherein the features describe the at least one object.

### Claim 9 (depends on 8)

The training device of claim 8, wherein the one or more processors are configured to fuse the plurality of features to fused features, and to train the object detection model using the fused features.

### Claim 10 (depends on 8)

The training device of claim 8, wherein the one or more processors are configured to train the object detection model by suppressing one or more selected features of the plurality of features, to fuse the plurality of features except for the selected features to fused features, and to train the object detection model using the fused features.

### Claim 11 (depends on 8)

The training device of claim 8, wherein the one or more processors are configured to:
determine, based on the object detection model, a detection uncertainty of the at least one object; and
train the object detection model using the determined detection uncertainty.

### Claim 12 (depends on 9)

The training device of claim 9, wherein the one or more processors are configured to:
generate, based on the object detection model, a class probability map using the fused features, wherein the class probability map comprises for each element of the plurality of elements of at least one sensor data item of the sensor data a probability of each class of a plurality of classes; and
train the object detection model using the generated class probability map.

### Claim 13 (depends on 9)

The training device of claim 9, wherein the one or more processors are configured to:
generate, based on the object detection model, at least one bounding box for at least one augmented sensor data item using the fused features, wherein the at least one bounding box comprises one or more elements of the one or more elements associated to the at least one object in the associated sensor data item; and
train the object detection model using the at least one bounding box.

### Claim 14 (depends on 10)

The training device of claim 10, wherein the one or more processors are configured to:
generate, based on the object detection model, a class probability map using the fused features, wherein the class probability map comprises for each element of the plurality of elements of at least one sensor data item of the sensor data a probability of each class of a plurality of classes;
generate, based on the object detection model, at least one bounding box for at least one augmented sensor data item using the fused features, wherein the at least one bounding box comprises one or more elements of the one or more elements associated to the at least one object in the associated sensor data item; and
train the object detection model using the generated class probability map and the generated at least one bounding box.

### Claim 15 (depends on 11)

The training device of claim 11, wherein the one or more processors are configured to:
generate training data, wherein the training data comprise information about the at least one object;
determine a training detection uncertainty for the at least one object using the training data;
train the object detection model using the determined training detection uncertainty and the determined detection uncertainty.

### Claim 16 (depends on 12)

The training device of claim 12, the one or more processors are configured to:
generate training data, wherein the training data comprise a training class for each element of the plurality of elements of the at least one sensor data item;
train the object detection model using the generated training data and the generated class probability map.

### Claim 17 (depends on 13)

The training device of claim 13, wherein the one or more processors are configured to:
generate training data, wherein the training data comprise at least one training bounding box, wherein the at least one training bounding box comprises each element of the one or more elements associated to the at least one object;
train the object detection model using the generated training data and the generated at least one bounding box.

### Claim 18

A device, comprising one or more processors, the one or more processors configured to:
generate, using a data augmentation model, augmented sensor data for sensor data, the sensor data provided by a plurality of sensors, wherein the augmented sensor data comprise error states of one or more sensors of the plurality of sensors providing the sensor data;
train an object detection model based on the augmented sensor data, wherein the object detection model comprises a plurality of feature extraction models, and wherein the one or more processors are configured to:
determine, based on the plurality of feature extraction models, a plurality of features for the augmented sensor data; and
train the object detection model based on the plurality of features, wherein the one or more processors are configured to train the object detection model by suppressing one or more selected features of the plurality of features, wherein the one or more selected features comprises all features associated to one or more selected sensors;
generate training data using the trained object detection model; and
train an additional model using the training data.

### Claim 19

A non-transitory computer-readable medium comprising instructions recorded thereon which, when executed by one or more processors, cause the one or more processors to:
generate, using a data augmentation model, augmented sensor data for sensor data, the sensor data provided by a plurality of sensors, wherein the augmented sensor data comprise error states of one or more sensors of the plurality of sensors providing the sensor data; and
train an object detection model based on the augmented sensor data, wherein the object detection model comprises a plurality of feature extraction models, and wherein the one or more processors are configured to:
determine, based on the plurality of feature extraction models, a plurality of features for the augmented sensor data; and
train the object detection model based on the plurality of features, wherein the one or more processors are configured to train the object detection model by suppressing one or more selected features of the plurality of features, wherein the one or more selected features comprises all features associated with one or more selected sensors.

### Claim 20

A non-transitory computer-readable medium comprising instructions recorded thereon which, when executed by one or more processors, cause the one or more processors to:
generate, using a data augmentation model, augmented sensor data for sensor data, the sensor data provided by a plurality of sensors, wherein the augmented sensor data comprise error states of one or more sensors of the plurality of sensors providing the sensor data; and
train an object detection model based on the augmented sensor data, wherein the object detection model comprises a plurality of feature extraction models, and wherein the one or more processors are configured to:
determine, based on the plurality of feature extraction models, a plurality of features for the augmented sensor data; and
train the object detection model based on the plurality of features, wherein the one or more processors are configured to train the object detection model by suppressing one or more selected features of the plurality of features, wherein the one or more selected features comprises all features associated with one or more selected sensors;
generate training data using the trained object detection model; and
train an additional model using the training data.
