# US 2022/0044073 A1 · Feature pyramids for object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044073 A1
- Document ID: US20220044073A1
- Status: Published application
- Kind: A1
- Application number: 17/386,879
- Filed: 2021-07-28
- Published: 2022-02-10
- Earliest claimed: 2020-08-07
- Assignees: Carnegie Mellon University
- Applicants: Carnegie Mellon University
- Inventors: Fangyi Chen; Chenchen Zhu; Zhiqiang Shen; Han Zhang; Marios Savvides
- Web page: https://patentgrep.com/patent/US20220044073A1

## CPC

- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F 18/2431: Multiple classes
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06V 10/52: Scale-space analysis, e.g. wavelet analysis (multi-scale boundary representations G06V10/42)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed herein is an improvement to prior art feature pyramids for general object detection that inserts a simple norm calibration (NC) operation between the feature pyramids and detection head to alleviate and balance the norm bias caused by feature pyramid network (FPN) and which leverages an enhanced multi-feature selective strategy (MS) during training to assign the ground-truth to one or more levels of the feature pyramid.

## Claims

### Claim 1

A framework for training an object detector comprising:
a feature pyramid;
a norm calibration layer, to generate a norm calibrated feature pyramid from the feature pyramid;
a localization subnetwork, coupled to the norm calibrated feature pyramid, the localization network generating a multi-level localization map;
a classification subnetwork, coupled to the norm calibrated feature pyramid, the classification subnetwork generating a multi-level heat map; and
a multi-selection module for collecting losses from the localization map and the heat map and assigning ground truth and ignored areas to one or more levels of the feature pyramid, based on the collected losses.

### Claim 2 (depends on 1)

The framework of claim 1 wherein the feature map has 5 levels, each level from bottom to top being half of the size of a next lowest level.

### Claim 3 (depends on 1)

The framework of claim 1 wherein the norm calibrated feature pyramid is identical in size and shape top the feature pyramid.

### Claim 4 (depends on 1)

The framework of claim 1 wherein the localization subnetwork is a 3×3 convolution layer having 4 filters.

### Claim 5 (depends on 1)

The framework of claim 1 wherein the classification subnetwork is a 3×3 convolution layer having a number of filters corresponding to a number of classification classes.

### Claim 6 (depends on 1)

The framework of claim 1 wherein the norm calibration layer is an l2 norm rescaling pattern.

### Claim 7 (depends on 6)

The framework of claim 6 wherein the axes of the norm calibration layer indexes features generated by the feature pyramid.

### Claim 8 (depends on 1)

The framework of claim 1 wherein the norm calibration layer generates a plurality of feature vectors, each feature vector being a vector across all channels of a single layer of the norm calibrated feature pyramid.

### Claim 9 (depends on 1)

The framework of claim 1 wherein an object in an input image is assigned to all levels of the localization and classification maps.

### Claim 10 (depends on 9)

The framework of claim 9 wherein a classification and regression loss is calculated for the object for each level of the localization and classification maps.

### Claim 11 (depends on 10)

The framework of claim 10 wherein the multi-selector module selects one or more levels of the feature pyramid to assign the object to based on a comparison of the losses for the object at each level of the localization and heat maps.

### Claim 12

A method for training an object detector comprising:
deriving a feature pyramid from an input image;
performing a norm calibration on the feature pyramid to generate a norm calibrated feature pyramid;
applying one or more subnetworks to the norm calibrated feature pyramid to generate one or more multi-level feature maps, the one or more feature maps having losses for each feature on each level of the feature maps; and
selecting one or more levels of the feature pyramid to assign detected objects to based on a comparison of the losses from each level of the one or more feature maps.

### Claim 13 (depends on 12)

The method of claim 12 wherein the feature map has 5 levels, each level from bottom to top being half of the size of a next lowest level.

### Claim 14 (depends on 12)

The method of claim 12 wherein the norm calibrated feature pyramid is identical in size and shape top the feature pyramid.

### Claim 15 (depends on 12)

The method of claim 12 wherein the one or more subnetworks includes a localization subnetwork comprising a 3×3 convolution layer having 4 filters.

### Claim 16 (depends on 12)

The method of claim 12 wherein the one or more subnetworks includes a classification subnetwork comprising a 3×3 convolution layer having a number of filters corresponding to a number of classification classes.

### Claim 17 (depends on 12)

The method of claim 12 wherein the norm calibration layer is an l2 norm rescaling pattern.

### Claim 18 (depends on 17)

The method of claim 17 wherein the axes of the norm calibration layer indexes features generated by the feature pyramid.

### Claim 19 (depends on 12)

The method of claim 12 wherein the norm calibration layer generates a plurality of feature vectors, each feature vector being a vector across all channels of a single layer of the norm calibrated feature pyramid.

### Claim 20 (depends on 12)

The method of claim 12 wherein an object in an input image is assigned to all levels of the localization and classification maps.

### Claim 21 (depends on 20)

The method of claim 20 wherein a classification and regression loss is calculated for the object for each level of the localization and classification maps.
