# US 11,443,522 B2 · Processing sensor information for object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,522 B2
- Document ID: US11443522B2
- Status: Granted
- Kind: B2
- Application number: 16/701,021
- Filed: 2019-12-02
- Published: 2020-06-04
- Granted: 2022-09-13
- Earliest claimed: 2018-11-30
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Bence Major; Daniel Hendricus Franciscus Fontijne; Ravi Teja Sukhavasi; Amin Ansari
- Web page: https://patentgrep.com/patent/US11443522B2

## CPC

- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06F 18/24: Classification techniques
- G06T 7/60: Analysis of geometric attributes
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G05D 1/0231: {using optical position detecting means (position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16)}
- G05D 1/0257: {using a radar (radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931)}
- G05D 1/0276: {using signals provided by a source external to the vehicle (involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058)}
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Methods of processing vehicle sensor information for object detection may include capturing generating a feature map based on captured sensor information, associating with each pixel of the feature map a prior box having a set of two or more width priors and a set of two or more height priors, determining a confidence value of each height prior and each width prior, outputting an indication of a detected object based on a highest confidence height prior and a highest confidence width prior, and performing a vehicle operation based on the output indication of a detected object. Embodiments may include determining for each pixel of the feature map one or more prior boxes having a center value, a size value, and a set of orientation priors, determining a confidence value for each orientation prior, and outputting an indication of the orientation of a detected object based on the highest confidence orientation.

## Claims

### Claim 1

A method of processing sensor information for object detection, comprising:
capturing, by a sensor, sensor information regarding an environment around the sensor;
generating, by a processor, a feature map based on the sensor information;
associating, by the processor, with each pixel of the feature map a prior box comprising:
a plurality of width priors that each include a width value,
a plurality of height priors that each include a height value, and
a plurality of orientation priors that each include an orientation value;
determining, by the processor, a confidence value of each height prior of the plurality of height priors, each width prior of the plurality of width priors, and each orientation prior of the plurality of orientation priors;
outputting, by the processor, an indication of a detected object based on a highest confidence height prior from among the plurality of height priors, a highest confidence width prior from among the plurality of width priors, and a highest confidence orientation prior from among the plurality of orientation priors; and
taking an action based on the output indication of a detected object.

### Claim 2 (depends on 1)

The method of claim 1, wherein taking an action based on the output indication of a detected object comprises:
performing, by the processor, a vehicle operation based on the output indication of a detected object.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
selecting the highest confidence height prior from the plurality of height priors; and
selecting the highest confidence width prior from the plurality of width priors.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
determining a height refinement value for each of the plurality of height priors; and
determining a width refinement value for each of the plurality of width priors.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
refining the highest confidence height prior with the height refinement value; and
refining the highest confidence width prior with the width refinement value.

### Claim 6 (depends on 5)

The method of claim 5, wherein outputting an indication of a detected object based on a highest confidence height prior from among the plurality of height priors and a highest confidence width prior from among the plurality of width priors comprises:
outputting the indication of a detected object comprising the refined highest confidence height prior and the refined highest confidence width prior.

### Claim 7 (depends on 1)

The method of claim 1, wherein the prior box associated with each pixel further comprises a center value.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
determining for each prior box a center value refinement; and
refining the center value with the determined center value refinement.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
determining for each prior box one or more object classifications; and
determining for each of the one or more object classifications a confidence value.

### Claim 10 (depends on 1)

The method of claim 1, wherein the output indication of a detected object comprises one or more of a height confidence value and a width confidence value.

### Claim 11

A vehicle control unit, comprising:
a processor configured with processor-executable instructions to perform operations comprising:
receiving sensor information from a sensor;
generating a feature map based on the sensor information;
associating with each pixel of the feature map a prior box comprising:
a plurality of width priors that each include a width value,
a plurality of height priors that each include a height value, and
a plurality of orientation priors that each include an orientation value;
determining a confidence value of each height prior of the plurality of height priors, each width prior of the plurality of width priors for each prior box, and each orientation prior of the plurality of orientation priors;
outputting an indication of a detected object based on a highest confidence height prior from among the plurality of height priors a highest confidence width prior from among the plurality of width priors, and a highest confidence orientation prior from among the plurality of orientation priors; and
taking an action based on the output indication of a detected object.

### Claim 12 (depends on 11)

The vehicle control unit of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that taking an action based on the output indication of a detected object comprises:
performing a vehicle operation based on the output indication of a detected object.

### Claim 13 (depends on 11)

The vehicle control unit system of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
selecting the highest confidence height prior from the plurality of height priors; and
selecting the highest confidence width prior from the plurality of width priors.

### Claim 14 (depends on 11)

The vehicle control unit of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
determining a height refinement value for each of the plurality of height priors; and
determining a width refinement value for each of the plurality of width priors.

### Claim 15 (depends on 14)

The vehicle control unit of claim 14, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
refining the highest confidence height prior with the height refinement value; and
refining the highest confidence width prior with the width refinement value.

### Claim 16 (depends on 15)

The vehicle control unit of claim 15, wherein the processor is configured with processor-executable instructions to perform operations such that outputting an indication of a detected object based on a highest confidence height prior from among the plurality of height priors and a highest confidence width prior from among the plurality of width priors comprises:
outputting the indication of a detected object comprising the refined highest confidence height prior and the refined highest confidence width prior.

### Claim 17 (depends on 11)

The vehicle control unit of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that each associated prior box further comprises a center value.

### Claim 18 (depends on 17)

The vehicle control unit of claim 17, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
determining for each prior box a center value refinement; and
refining the center value with the determined center value refinement.

### Claim 19 (depends on 11)

The vehicle control unit of claim 11, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
determining for each prior box one or more object classifications; and
determining for each of the one or more object classifications a confidence value.

### Claim 20 (depends on 11)

The vehicle control unit of claim 11, wherein the processor is configured with processor-executable instructions to perform operations such that the output indication of a detected object comprises one or more of a height confidence value and a width confidence value.

### Claim 21

A method of processing vehicle sensor information for object detection, comprising:
capturing, by a sensor, sensor information regarding an environment around the sensor;
generating, by a processor, a feature map based on the sensor information;
associating, by the processor, with each pixel of the feature map, one or more prior boxes each comprising a center value, a size value, and a plurality of orientation priors that each include an orientation value;
determining, by the processor, a confidence value for each orientation prior of the plurality of orientation priors;
outputting, by the processor, an indication of a detected object based on the highest confidence orientation; and
performing, by the processor, a vehicle operation based on the output indication of a detected object.

### Claim 22 (depends on 21)

The method of claim 21, further comprising:
selecting for each prior box a highest confidence orientation prior from among the plurality of orientation priors.

### Claim 23 (depends on 21)

The method of claim 21, further comprising:
determining an orientation refinement for each of the plurality of orientation priors.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
refining an orientation value for each prior box based on the determined orientation refinement.

### Claim 25 (depends on 24)

The method of claim 24, further comprising:
rotating each prior box based on the refined orientation value.

### Claim 26 (depends on 21)

The method of claim 21, wherein the output indication of a detected object comprises an orientation confidence value.

### Claim 27

A vehicle control unit, comprising:
a processor configured with processor-executable instructions to perform operations comprising:
receiving sensor information regarding an environment around a sensor;
generating, by a processor, a feature map based on the sensor information;
associating, by the processor, with each pixel of the feature map, one or more prior boxes each comprising a center value, a size value, and a plurality of orientation priors that each include an orientation value;
determining, by the processor, a confidence value for each orientation prior of the plurality of orientation priors;
outputting, by the processor, an indication of a detected object based on the highest confidence orientation; and
performing, by the processor, a vehicle operation based on the output indication of a detected object.

### Claim 28 (depends on 27)

The vehicle control unit of claim 27, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
selecting for each prior box a highest confidence orientation prior from among the plurality of orientation priors.

### Claim 29 (depends on 27)

The vehicle control unit of claim 27, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
determining an orientation refinement for each of the plurality of orientation priors; and
refining an orientation value for each prior box based on the determined orientation refinement.

### Claim 30 (depends on 29)

The vehicle control unit of claim 29, wherein the processor is configured with processor-executable instructions to perform operations further comprising:
rotating each prior box based on the refined orientation value.