# US 2019/0171903 A1 · Optimizations for Dynamic Object Instance Detection, Segmentation, and Structure Mapping

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171903 A1
- Document ID: US20190171903A1
- Status: Published application
- Kind: A1
- Application number: 15/971,930
- Filed: 2018-05-04
- Published: 2019-06-06
- Earliest claimed: 2017-12-03
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Peter Vajda; Peizhao Zhang; Fei Yang; Yanghan Wang
- Web page: https://patentgrep.com/patent/US20190171903A1

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/2113: by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
- G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/11: Region-based segmentation
- G06T 7/75: {involving models}
- G06V 10/764: using classification, e.g. of video objects
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20104: Interactive definition of region of interest [ROI]
- G06T 2207/30196: Human being; Person
- G06T 2210/12: Bounding box
- G06V 20/20: in augmented reality scenes
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one embodiment, a system may access an image and generate a feature map for the image using a neural network. The system may identify regions of interest in the feature map. Regional feature maps may be generated for the regions of interest, respectively. Each of the regional feature maps has a first, a second, and a third dimension. The system may generate a first combined regional feature map by combining the regional feature maps. The combined regional feature map has a first, a second, and a third dimension. The system may generate a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers. The system may generate, for each of the regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.

## Claims

### Claim 1

A method comprising, by a computing system:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.

### Claim 3 (depends on 2)

The method of claim 2, wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.

### Claim 4 (depends on 3)

The method of claim 3,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size;
wherein the first dimension or the second dimension corresponds to channel size.

### Claim 5 (depends on 1)

The method of claim 1, wherein the first combined regional feature map includes the plurality of regional feature maps with paddings inserted between adjacent pairs of the plurality of regional feature maps.

### Claim 6 (depends on 5)

The method of claim 5, wherein a size of the padding between each adjacent pair of the plurality of regional feature maps is at least as wide as a kernel size used by the one or more convolutional layers.

### Claim 7 (depends on 1)

The method of claim 1, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors.

### Claim 8 (depends on 1)

The method of claim 1, wherein the information associated with the object instance is an instance segmentation mask, a keypoint mask, or a bounding box.

### Claim 9

A system comprising: one or more processors and one or more computer-readable non-transitory storage media coupled to one or more of the processors, the one or more computer-readable non-transitory storage media comprising instructions operable when executed by one or more of the processors to cause the system to perform operations comprising:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.

### Claim 10 (depends on 9)

The system of claim 9,
wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.

### Claim 11 (depends on 10)

The system of claim 10, wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.

### Claim 12 (depends on 11)

The system of claim 11,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size; and
wherein the first dimension or the second dimension corresponds to channel size.

### Claim 13 (depends on 9)

The system of claim 9, wherein the first combined regional feature map includes the plurality of regional feature maps with paddings inserted between adjacent pairs of the plurality of regional feature maps.

### Claim 14 (depends on 13)

The system of claim 13, wherein a size of the padding between each adjacent pair of the plurality of regional feature maps is at least as wide as a kernel size used by the one or more convolutional layers.

### Claim 15 (depends on 9)

The system of claim 9, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors.

### Claim 16

One or more computer-readable non-transitory storage media embodying software that is operable when executed to cause one or more processors to perform operations comprising:
accessing an image;
generating a feature map for the image using a neural network;
identifying a plurality of regions of interest in the feature map;
generating a plurality of regional feature maps for the plurality of regions of interest, respectively, wherein each of the plurality of regional feature maps has a first dimension, a second dimension, and a third dimension;
generating a first combined regional feature map by combining the plurality of regional feature maps, wherein the combined regional feature map has a first dimension, a second dimension, and a third dimension;
generating a second combined regional feature map by processing the first combined regional feature map using one or more convolutional layers; and
generating, for each of the plurality of regions of interest, information associated with an object instance based on a portion of the second combined regional feature map associated with that region of interest.

### Claim 17 (depends on 16)

The media of claim 16,
wherein the first dimension and the second dimension of the first combined regional feature map are equal to the first dimension and the second dimension of each of the plurality of regional feature maps, respectively.

### Claim 18 (depends on 17)

The media of claim 17, wherein the third dimension of the first combined regional feature map is equal to or larger than a combination of the respective third dimensions of the plurality of regional feature maps.

### Claim 19 (depends on 18)

The media of claim 18,
wherein the third dimension of each of the plurality of regional feature maps corresponds to height size or width size;
wherein the first dimension or the second dimension corresponds to channel size.

### Claim 20 (depends on 16)

The media of claim 16, wherein the processing of the first combined regional feature map is performed using a neural processing engine configured for performing convolutional operations on three-dimensional tensors.
