# US 2021/0264557 A1 · System and method for real-time, simultaneous object detection and semantic segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264557 A1
- Document ID: US20210264557A1
- Status: Published application
- Kind: A1
- Application number: 16/802,078
- Filed: 2020-02-26
- Published: 2021-08-26
- Earliest claimed: 2020-02-26
- Assignees: Beijing Jingdong Shangke Information Technology Co., Ltd.; JD.com American Technologies Corporation
- Applicants: Beijing Jingdong Shangke Information Technology Co., Ltd.; JD.com American Technologies Corporation
- Inventors: Hongda Mao; Wei Xiang; Chumeng Lyu; Weidong Zhang
- Web page: https://patentgrep.com/patent/US20210264557A1

## CPC

- G06T 7/73: using feature-based methods
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/11: Region-based segmentation
- G06V 10/451: {with interaction between the filter responses, e.g. cortical complex cells}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06N 3/08: Learning methods
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

System and method for simultaneous object detection and semantic segmentation. The system includes a computing device. The computing device has a processor and a non-volatile memory storing computer executable code. The computer executable code, when executed at the processor, is configured to: receive an image of a scene; process the image using a neural network backbone to obtain a feature map; process the feature map using an object detection module to obtain object detection result of the image; and process the feature map using a semantic segmentation module to obtain semantic segmentation result of the image. The object detection module and the semantic segmentation module are trained using a same loss function comprising an object detection component and a semantic segmentation component.

## Claims

### Claim 1

A system for object detection and semantic segmentation, the system comprising a computing device, the computing device comprising a processor and a non-volatile memory storing computer executable code, wherein the computer executable code, when executed at the processor, is configured to:
receive an image of a scene;
process the image using a neural network backbone to obtain a feature map;
process the feature map using an object detection module to obtain object detection result of the image; and
process the feature map using a semantic segmentation module to obtain semantic segmentation result of the image,
wherein the object detection module and the semantic segmentation module are trained using a same loss function comprising an object detection component and a semantic segmentation component; and
wherein the neural network backbone comprises a ResNet18 backbone truncated away from its 4th block.

### Claim 2 (depends on 1)

The system of claim 1, wherein training data for the object detection module and the semantic segmentation module comprises: a training image, at least one bounding box defined in the training image, label of the at least one bounding box, and mask of the training image.

### Claim 3 (depends on 1)

The system of claim 1, wherein the object detection module is a single shot detector (SSD).

### Claim 4 (depends on 1)

The system of claim 1, wherein the object detection module consists of sequentially:
five convolution layers;
a detection layer; and
a non-maximum suppression (NMS) layer.

### Claim 5 (depends on 4)

The system of claim 4, wherein for a 512×512 resolution of the image, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers respectively add 16×16 information, 8×8 information, 4×4 information, 2×2 information, and 1×1 information to the detection layer.

### Claim 6 (depends on 1)

The system of claim 1, wherein the semantic segmentation module is a pyramid pooling module.

### Claim 7 (depends on 1)

The system of claim 1, wherein the semantic segmentation module consists sequentially:
a pooling module pooling the feature map to obtain pooled features at different sizes;
a plurality of convolution layers each convoluting one of the pooled features at different sizes to obtain convoluted features at different sizes;
an upsample module receiving the convoluted features at different sizes to obtain upsampled feature;
a concatenation layer receiving the upsampled feature and feature from the neural network backbone to obtain concatenated feature; and
a convolution layer convoluting the concatenated feature to obtain per-pixel prediction as the semantic segmentation result.

### Claim 8 (depends on 7)

The system of claim 7, wherein the pooled features are at sizes of 1×1, 2×2, 3×3, and 6×6.

### Claim 9 (depends on 1)

The system of claim 1, wherein the computer executable code is further configured to control an operative device in the scene based on the object detection result and the semantic segmentation result.

### Claim 10

A method for object detection and semantic segmentation, comprising:
receiving, by a computing device, an image of a scene;
processing, by the computing device, the image using a neural network backbone to obtain a feature map;
processing, by the computing device, the feature map using an object detection module to obtain object detection result of the image; and
processing, by the computing device, the feature map using a semantic segmentation module to obtain semantic segmentation result of the image,
wherein the object detection module and the semantic segmentation module are trained using a same loss function comprising an object detection component and a semantic segmentation component; and
wherein the neural network backbone comprises a ResNet18 backbone truncated away from its 4th block.

### Claim 11 (depends on 10)

The method of claim 10, wherein training data for the object detection module and the semantic segmentation module comprises: a training image, at least one bounding box defined in the training image, label of the at least one bounding box, and mask of the training image.

### Claim 12 (depends on 10)

The method of claim 10, wherein the object detection module is a single shot detector.

### Claim 13 (depends on 10)

The method of claim 10, wherein the object detection module consists of sequentially:
five convolution layers;
a detection layer; and
a non-maximum suppression (NMS) layer.

### Claim 14 (depends on 13)

The method of claim 13, wherein for a 512×512 resolution of the image, the neural network backbone convolutionally adds 64×64 information and 32×32 information to the detection layer, and the five convolutional layers respectively add 16×16 information, 8×8 information, 4×4 information, 2×2 information, and 1×1 information to the detection layer.

### Claim 15 (depends on 10)

The method of claim 10, wherein the semantic segmentation module is a pyramid pooling module.

### Claim 16 (depends on 10)

The method of claim 10, wherein the semantic segmentation module consists sequentially:
a pooling module pooling the feature map to obtain pooled features at different sizes;
a plurality of convolution layers each convoluting one of the pooled features at different sizes to obtain convoluted features at different sizes;
an upsample module receiving the convoluted features at different sizes to obtain upsampled feature;
a concatenation layer receiving the upsampled feature and feature from the neural network backbone to obtain concatenated feature; and
a convolution layer convoluting the concatenated feature to obtain the semantic segmentation result.

### Claim 17 (depends on 10)

The method of claim 10, further comprising: controlling an operative device in the scene based on the object detection result and the semantic segmentation result.

### Claim 18

A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to:
receive an image of a scene;
process the image using a neural network backbone to obtain a feature map;
process the feature map using an object detection module to obtain object detection result of the image; and
process the feature map using a semantic segmentation module to obtain semantic segmentation result of the image,
wherein the object detection module and the semantic segmentation module are trained using a same loss function comprising an object detection component and a semantic segmentation component; and
wherein the neural network backbone comprises a ResNet18 backbone truncated away from its 4th block.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the object detection module consists of sequentially:
five convolution layers;
a detection layer; and
a non-maximum suppression (NMS) layer.

### Claim 20 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the semantic segmentation module consists sequentially:
a pooling module pooling the feature map to obtain pooled features at different sizes;
a plurality of convolution layers each convoluting one of the pooled features at different sizes to obtain convoluted features at different sizes;
an upsample module receiving the convoluted features at different sizes to obtain upsampled feature;
a concatenation layer receiving the upsampled feature and feature from the neural network backbone to obtain concatenated feature; and
a convolution layer convoluting the concatenated feature to obtain per-pixel prediction as the semantic segmentation result.
