# US 2023/0266771 A1 · Restricted multi-scale inference for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0266771 A1
- Document ID: US20230266771A1
- Status: Published application
- Kind: A1
- Application number: 18/114,531
- Filed: 2023-02-27
- Published: 2023-08-24
- Earliest claimed: 2018-06-20
- Assignees: Zoox, Inc.
- Applicants: Zoox, Inc.
- Inventors: Sarah Tariq; Kratarth Goel; James William Vaisey Philbin
- Web page: https://patentgrep.com/patent/US20230266771A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.

## Claims

### Claim 1

(canceled)

### Claim 2

A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
providing, as input to a first machine-learning (ML) model associated with a first size range, an image;
determining, by the first ML model and based at least in part on the image and the first size range, a first region of interest (ROI);
suppressing the first ROI, wherein suppressing the first ROI comprises determining that the first ROI is associated with a first size that is outside the first size range; and
receiving, as a second output from the first ML model, a second ROI associated with an object or a first indication that a dimension of the object is outside the first size range.

### Claim 3 (depends on 2)

The system as claim 2 recites, the operations further comprising:
providing, as input to a second ML model associated with a second size range, the image;
determining, by the second ML model and based at least in part on the image and the second size range, a third ROI;
suppressing the third ROI, wherein suppressing the third ROI comprises determining that at least a portion of the third ROI is associated with a second size that is outside the second size range; and
receiving, from the second ML model, a fourth ROI associated with the object or a second indication that the dimension of the object is outside the second size range.

### Claim 4 (depends on 3)

The system as claim 3 recites, wherein:
the first ML model outputs a fifth ROI having a third size within the first size range, based at least in part on a first accuracy of the first ML model associated with the first size range; and
the second ML model outputs a sixth ROI having a fourth size within the second size range, based at least in part on a second accuracy of the second ML model associated with the second size range.

### Claim 5 (depends on 2)

The system as claim 2 recites, the operations further comprising:
determining the first size range for the first ML model based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions;
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first size range based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment.

### Claim 6 (depends on 2)

The system as claim 2 recites, the operations further comprising:
receiving a batch of images, wherein the batch of images comprises a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification; and
training the first ML model based at least in part on providing the batch of images as input to the first ML model,
wherein the first predefined number of images and the second predefined number of images are based at least in part on a confidence score associated with the first ML model or a second ML model.

### Claim 7

One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
providing, as input to a first machine-learning (ML) model associated with a first size range, an image;
determining, by the first ML model and based at least in part on the image and the first size range, a first region of interest (ROI);
suppressing the first ROI, wherein suppressing the first ROI comprises determining that the first ROI is associated with a first size that is outside the first size range; and
receiving, as a second output from the first ML model, a second ROI associated with an object or a first indication that a dimension of the object is outside the first size range.

### Claim 8 (depends on 7)

The one or more non-transitory computer-readable media as claim 7 recites, wherein receiving the first indication that the dimension of the object is outside the first size range is based at least in part on determining that the first ROI includes all of a plurality of ROIs.

### Claim 9 (depends on 7)

The one or more non-transitory computer-readable media as claim 7 recites, wherein the operations further comprise:
providing, as input to a second ML model associated with a second size range, the image;
determining, by the second ML model, a third ROI;
suppressing the third ROI, wherein suppressing the third ROI comprises determining that at least a portion of the third ROI is associated with a second size that is outside the second size range; and
receiving, from the second ML model, a fourth ROI associated with the object or a second indication that the dimension of the object is outside the second size range.

### Claim 10 (depends on 9)

The one or more non-transitory computer-readable media as claim 9 recites, wherein a fifth ROI corresponding to the object is received from the first ML model or the second ML model, based at least in part on the dimension of the object in the image, the first size range, and the second size range.

### Claim 11 (depends on 9)

The one or more non-transitory computer-readable media as claim 9 recites, wherein the operations further comprise:
determining, by the first ML model, a fifth ROI having a third size within the first size range, based at least in part on a first accuracy of the first ML model associated with the first size range; and
determining, by the second ML model, a sixth ROI having a fourth size within the second size range, based at least in part on a second accuracy of the second ML model associated with the second size range.

### Claim 12 (depends on 9)

The one or more non-transitory computer-readable media as claim 9 recites, wherein the operations further comprise:
generating, based at least in part on at least one of the second ROI or the fourth ROI, a trajectory for controlling motion of an autonomous vehicle.

### Claim 13 (depends on 9)

The one or more non-transitory computer-readable media as claim 9 recites, wherein the operations further comprise:
selecting the first size range and the second size range based at least in part on a machine learned model.

### Claim 14 (depends on 7)

The one or more non-transitory computer-readable media as claim 7 recites, wherein the operations further comprise determining the first size range for the first ML model based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions;
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first size range based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment.

### Claim 15 (depends on 7)

The one or more non-transitory computer-readable media as claim 7 recites, wherein the operations further comprise:
receiving a batch of images, wherein the batch of images comprises a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification; and
training the first ML model based at least in part on providing the batch of images as input to the first ML model,
wherein the first predefined number and the second predefined number are based at least in part on a confidence score associated with the first ML model or a second ML model.

### Claim 16

A method, comprising:
providing, as input to a first machine-learning (ML) model associated with a first size range, an image;
determining, by the first ML model and based at least in part on the image, a first regions of interest (ROIs);
suppressing a first ROI, wherein suppressing the first ROI comprises determining that the first ROI is associated with a first size that is outside the first size range; and
receiving a second ROI associated with an object or a first indication that a dimension of the object is outside the first size range.

### Claim 17 (depends on 16)

The method as claim 16 recites, wherein receiving the first indication that the dimension of the object is outside the first size range is based at least in part on determining that the first ROI includes all of a plurality of ROIs.

### Claim 18 (depends on 16)

The method as claim 16 recites, further comprising:
providing, as input to a second ML model associated with a second size range, the image, wherein providing the image to the second ML model occurs substantially simultaneously as providing the image to the first ML model;
determining, by the second ML model, a third ROI;
suppressing the third ROI, wherein suppressing the third ROI comprises determining that at least a portion of the third ROI is associated with a second size that is outside the second size range; and
receiving, from the second ML model, a fourth ROI associated with the object or a second indication that the dimension of the object is outside the second size range.

### Claim 19 (depends on 18)

The method as claim 18 recites, further comprising:
determining, by the first ML model, a fifth ROI having a third size within the first size range, based at least in part on a first accuracy of the first ML model associated with the first size range; and
determining, by the second ML model, a sixth ROIs having a fourth size within the second size range, based at least in part on a second accuracy of the second ML model associated with the second size range.

### Claim 20 (depends on 16)

The method as claim 16 recites, further comprising:
determining the first size range for the first ML model based at least in part on:
training the first ML model, wherein the training comprises:
providing, as input to the first ML model, test images that include test objects associated with areas defined by reference regions;
determining, by the first ML model and based at least in part on the test images, multiple ROIs;
determining degrees of alignment of the multiple ROIs to an area of the areas defined by the reference regions; and
determining the first size range based at least in part on identifying a span of object sizes that corresponds to a portion of the degrees of alignment that meet or exceed a threshold degree of alignment.

### Claim 21 (depends on 16)

The method as claim 16 recites, further comprising:
receiving a batch of images, wherein the batch of images comprises a first predefined number of images that are associated with a first object classification and a second predefined number of images that are associated with a second object classification; and
training the first ML model based at least in part on providing the batch of images as input to the first ML model,
wherein the first predefined number of images and the second predefined number of images are based at least in part on a confidence score associated with the first ML model or a second ML model.
