# US 11,908,160 B2 · Method and apparatus for context-embedding and region-based object detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,160 B2
- Document ID: US11908160B2
- Status: Granted
- Kind: B2
- Application number: 17/283,276
- Filed: 2018-10-12
- Published: 2021-12-09
- Granted: 2024-02-20
- Earliest claimed: 2018-10-12
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Jing Nie
- Web page: https://patentgrep.com/patent/US11908160B2

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method of detecting an object in an image using a convolutional neural network (CNN) includes generating, based on the image, a plurality of reference feature maps and a corresponding feature pyramid including a plurality of final feature maps; obtaining a proposed region of interest (ROI); generating at least a first context ROI having an area larger than an area of the proposed ROI; assigning the proposed ROI and the first context ROI to a first and second final feature maps having different sizes; extracting, by performing ROI pooling, a first set of features from the first final feature map using the proposed ROI and a second set of features from the second final feature map using the first context ROI; and determining, based on the first and second sets of extracted features, at least one of a location of the object and a class of the object.

## Claims

### Claim 1

A method of detecting an object in an image using a convolutional neural network (CNN), the method comprising:
generating, by the CNN, a plurality of reference feature maps based on the image;
generating a feature pyramid including a plurality of final feature maps corresponding, respectively, to the plurality of reference feature maps, wherein respective ones of the plurality of final feature maps are associated with different levels of the feature pyramid;
obtaining a proposed region of interest (ROI);
generating at least a first context ROI based on the proposed ROI such that an area of the first context ROI is larger than an area of the proposed ROI;
assigning the proposed ROI to a first final feature map, associated with one level of the different levels of the feature pyramid, from among the plurality of final feature maps;
assigning the first context ROI to a second final feature map, associated with another level of the different levels of the feature pyramid, from among the plurality of final feature maps, a size of the first final feature map being different than a size of the second final feature map;
extracting a first set of features from the first final feature map associated with the one level of the different levels of the feature pyramid by performing an ROI pooling operation on the first final feature map using the proposed ROI;
extracting a second set of features from the second final feature map associated with the other level of the different levels of the feature pyramid by performing an ROI pooling operation on the second final feature map using the first context ROI;
applying a concatenation of the first set of features and the second set of features to a squeeze-and-excitation block (SEB); and
determining, based on an output of the SEB, one or more probability values associated with a location of the object with respect to the image and a class of the object.

### Claim 2 (depends on 1)

The method of claim 1, wherein the feature pyramid is generated based on the plurality of reference feature maps in accordance with a feature pyramid network (FPN) architecture.

### Claim 3 (depends on 1)

The method of claim 1, wherein the area of the first context is 22 times the area of the proposed ROI.

### Claim 4 (depends on 1)

The method of claim 1,
wherein the determining includes determining, based on the concatenated sets of extracted features, at least one of the location of the object with respect to the image and the class of the object.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating a second context ROI based on the proposed ROI such that an area of the second context ROI is larger than an area of the first context ROI;
assigning the second context ROI to a third final feature map from among the plurality of final feature maps, a size of the third final feature map being different than the sizes of the first and second final feature maps; and
extracting a third set of features from the first final feature map by performing ROI pooling on the first final feature map using the second context ROI,
wherein the determining includes determining, based on the first, second and third sets of extracted features, at least one of the location of the object with respect to the image and the class of the object.

### Claim 6 (depends on 5)

The method of claim 5, wherein the feature pyramid is generated based on the plurality of reference feature maps in accordance with a feature pyramid network (FPN) architecture.

### Claim 7 (depends on 5)

The method of claim 5, wherein the area of the first context ROI is 22 times the area of the proposed ROI, and the area of the second context ROI is 42 times the area of the area of the proposed ROI.

### Claim 8 (depends on 5)

The method of claim 5, further comprising:
concatenating the first, second and third sets of extracted features,
wherein the determining includes determining, based on the concatenated sets of extracted features, at least one of the location of the object with respect to the image and the class of the object.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
applying the concatenated sets of extracted features to a squeeze-and-excitation block (SEB),
wherein the location of the object with respect to the image and the class of the object is determined based on an output of the SEB.

### Claim 10

A non-transitory computer-readable medium comprising program instructions for causing an apparatus to perform at least the following:
generating, by a convolutional neural network (CNN), a plurality of reference feature maps based on an image that includes an object;
generating a feature pyramid including a plurality of final feature maps corresponding, respectively, to the plurality of reference feature maps, wherein respective ones of the plurality of final feature maps are associated with different levels of the feature pyramid;
obtaining a proposed region of interest (ROI);
generating at least a first context ROI based on the proposed ROI such that an area of the first context ROI is larger than an area of the proposed ROI;
assigning the proposed ROI to a first final feature map, associated with one level of the different levels of the feature pyramid, from among the plurality of final feature maps;
assigning the first context ROI to a second final feature map, associated with another level of the different levels of the feature pyramid, from among the plurality of final feature maps, a size of the first final feature map being different than a size of the second final feature map;
extracting a first set of features from the first final feature map associated with the one level of the different levels of the feature pyramid by performing an ROI pooling operation on the first final feature map using the proposed ROI;
extracting a second set of features from the second final feature map associated with the other level of the different levels of the feature pyramid by performing an ROI pooling operation on the second final feature map using the first context ROI;
applying a concatenation of the first set of features and the second set of features to a squeeze-and-excitation block (SEB); and
determining, based on an output of the SEB, one or more probability values associated with a location of the object with respect to the image and a class of the object.

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the feature pyramid is generated based on the plurality of reference feature maps in accordance with a feature pyramid network (FPN) architecture.

### Claim 12 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the area of the first context ROI is 22 times the area of the proposed ROI.

### Claim 13 (depends on 10)

The non-transitory computer-readable medium of claim 10,
wherein the determining includes determining, based on the concatenated sets of extracted features, at least one of the location of the object with respect to the image and the class of the object.

### Claim 14

An apparatus comprising:
at least one processor; and
at least one memory including computer program code,
the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform:
generating, by a convolutional neural network (CNN), a plurality of reference feature maps based on an image that includes an object;
generating a feature pyramid including a plurality of final feature maps corresponding, respectively, to the plurality of reference feature maps, wherein respective ones of the plurality of final feature maps are associated with different levels of the feature pyramid;
obtaining a proposed region of interest (ROI);
generating at least a first context ROI based on the proposed ROI such that an area of the first context ROI is larger than an area of the proposed ROI;
assigning the proposed ROI to a first final feature map, associated with one level of the different levels of the feature pyramid, from among the plurality of final feature maps;
assigning the first context ROI to a second final feature map, associated with another level of the different levels of the feature pyramid, from among the plurality of final feature maps, a size of the first final feature map being different than a size of the second final feature map;
extracting a first set of features from the first final feature map associated with the one level of the different levels of the feature pyramid by performing an ROI pooling operation on the first final feature map using the proposed ROI;
extracting a second set of features from the second final feature map associated with the other level of the different levels of the feature pyramid by performing an ROI pooling operation on the second final feature map using the first context ROI;
applying a concatenation of the first set of features and the second set of features to a squeeze-and-excitation block (SEB); and
determining, based on an output of the SEB, one or more probability values associated with a location of the object with respect to the image and a class of the object.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the feature pyramid is generated based on the plurality of reference feature maps in accordance with a feature pyramid network (FPN) architecture.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the area of the first context ROI is twice the area of the proposed ROI.

### Claim 17 (depends on 14)

The apparatus of claim 14,
wherein the determining includes determining, based on the concatenated sets of extracted features, at least one of the location of the object with respect to the image and the class of the object.