# US 2023/0274532 A1 · Learning Data Augmentation Strategies for Object Detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274532 A1
- Document ID: US20230274532A1
- Status: Published application
- Kind: A1
- Application number: 18/313,772
- Filed: 2023-05-08
- Published: 2023-08-31
- Earliest claimed: 2018-05-18
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Jon Shlens; Ekin Dogus Cubuk; Quoc Le; Tsung-Yi Lin; Barret Zoph; Golnaz Ghiasi
- Web page: https://patentgrep.com/patent/US20230274532A1

## CPC

- G06V 10/772: Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/24: Classification techniques
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and
for each of one or more iterations:
applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images;
training a machine-learned object detection model based at least in part on the one or more augmented images; and
after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model.

### Claim 22 (depends on 21)

The computing system of claim 21, wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises selecting the one or more augmentation operations from a defined search space that includes a plurality of available augmentation operations.

### Claim 23 (depends on 22)

The computing system of claim 22, wherein the plurality of available augmentation operations comprises one or more color operations that modify color channel data associated with at least a portion of the one or more training images.

### Claim 24 (depends on 23)

The computing system of claim 23, wherein the one or more color operations comprise one or more of:
an auto contrast operation;
an equalize operation;
a solarize operation;
a posterize operation;
a contrast operation;
a color balance operation;
a brightness operation;
a sharpness operation; and
a cutout operation.

### Claim 25 (depends on 22)

The computing system of claim 22, wherein the plurality of available augmentation operations comprises one or more geometric distortion operations that geometrically distort at least a portion of the one or more training images.

### Claim 26 (depends on 25)

The computing system of claim 25, wherein the one or more geometric distortion operations comprise one or more of:
a shear operation;
a translate operation;
a rotate operation;
a flipping operation; and
a shift bounding shape operation.

### Claim 27 (depends on 22)

The computing system of claim 22, wherein the plurality of available augmentation operations comprises:
one or more operations that augment the one or more training images without modification of the location of the bounding shape or the location of the target object within the one or more training images; and
one or more operations that modify the location of the bounding shape and the location of the target object within the one or more training images.

### Claim 28 (depends on 21)

The computing system of claim 21, wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
selecting a respective probability of performance for each of the one or more augmentation operations; and
performing each of the one or more augmentation operations with probability equal to the respective probability of performance.

### Claim 29 (depends on 21)

The computing system of claim 21, wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
selecting a respective probability that each of the one or more augmentation operations will be applied only to the bounding shape of the one or more training images; and
applying each augmentation operation to only the bounding shape of the one or more training images with probability equal to the respective probability.

### Claim 30 (depends on 21)

The computing system of claim 21, wherein applying the one or more augmentation operations only to the bounding shape of the one or more training images comprises:
selecting a respective augmentation magnitude for at least one of the augmentation operations; and
performing the at least one of the augmentation operations according to the respective augmentation magnitude.

### Claim 31 (depends on 30)

The computing system of claim 30, wherein selecting the respective augmentation magnitude for at least one of the augmentation operations comprises selecting the respective augmentation magnitude for at least one of the augmentation operations from a respective set of discrete and operation-specific available magnitudes, wherein the set of discrete and operation-specific available magnitudes comprise user-selected hyperparameters.

### Claim 32 (depends on 21)

The computing system of claim 21, wherein the computing system further comprises a controller model comprising a first neural network, and wherein the operations further comprise:
for each of the one or more iterations:
determining a reward based at least in part on the one or more performance characteristics; and
backpropagating a reward function through the first neural network to modify one or more parameters of the first neural network.

### Claim 33 (depends on 21)

The computing system of claim 21, wherein the computing system further comprises a controller model comprising a first neural network configured to select one or more augmentation operations through performance of evolutionary mutations, and wherein the operations further comprise, for each of the one or more iterations, determining whether to retain or discard the one or more augmentation operations based at least in part on the one or more performance characteristics.

### Claim 34 (depends on 21)

The computing system of claim 21, wherein training the machine-learned object detection model based at least in part on the one or more augmented images comprises:
evaluating, for each augmented image, a loss function that compares a predicted location for the bounding shape of the augmented image that was predicted by the machine-learned object detection model based on the augmented image to a ground truth location for the bounding shape; and
backpropagating the loss function through the machine-learned object detection model.

### Claim 35 (depends on 21)

The computing system of claim 21, wherein, for each iteration, a number of augmentation operations in the one or more augmentation operations is selected by a controller model comprising a first neural network.

### Claim 36 (depends on 35)

The computing system of claim 35, wherein the first neural network of the controller model comprises a recurrent neural network.

### Claim 37 (depends on 21)

The computing system of claim 21, wherein, for each iteration, a number of augmentation operations in the one or more augmentation operations is a user-selected hyperparameter.

### Claim 38 (depends on 21)

The computing system of claim 21, wherein the operations further comprise:
obtaining a second training dataset comprising a second plurality of training images;
applying, on one or more second training images of the second plurality of training images, at least one of the one or more augmentation operations only to the bounding shape of the one or more second training images in at least one of the one or more iterations; and
training a second machine-learned object detection model based at least in part on the one or more second training images.

### Claim 39

A computer-implemented method, comprising:
accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and
for each of one or more iterations:
applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images;
training a machine-learned object detection model based at least in part on the one or more augmented images; and
after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model.

### Claim 40

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
accessing a training dataset that comprises a plurality of training images, wherein one or more training images of the plurality of training images has been annotated with a bounding shape, and wherein a location of the bounding shape for the one or more training images corresponds to a location of a target object within the one or more training images; and
for each of one or more iterations:
applying one or more augmentation operations only to the bounding shape of the one or more training images to generate one or more augmented images;
training a machine-learned object detection model based at least in part on the one or more augmented images; and
after training the machine-learned object detection model, evaluating one or more performance characteristics of the machine-learned object detection model.
