# US 2020/0286383 A1 · Multi-task perception network with applications to scene understanding and advanced driver-assistance system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286383 A1
- Document ID: US20200286383A1
- Status: Published application
- Kind: A1
- Application number: 16/787,727
- Filed: 2020-02-11
- Published: 2020-09-10
- Earliest claimed: 2019-03-07
- Assignees: NEC Laboratories America, Inc.
- Applicants: NEC Laboratories America, Inc.
- Inventors: Quoc-Huy Tran; Samuel Schulter; Paul Vernaza; Buyu Liu; Pan Ji; Yi-Hsuan Tsai; Manmohan Chandraker
- Web page: https://patentgrep.com/patent/US20200286383A1

## CPC

- G08G 1/162: {event-triggered}
- B60W 10/18: including control of braking systems
- B60W 10/20: including control of steering systems
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G08G 1/166: {for active traffic, e.g. moving vehicles, pedestrians, bikes}
- G08G: TRAFFIC CONTROL SYSTEMS (guiding railway traffic, ensuring the safety of railway traffic B61L; arrangement of road signs or traffic signals E01F9/00; radar or analogous systems, sonar systems, lidar systems specially adapted for traffic control G01S13/91, G01S15/88, G01S17/88; {radar or analogous systems, sonar systems, lidar systems specially adapted for anti-collision purposes G01S13/93, G01S15/93, G01S17/93})
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method is provided in an Advanced Driver-Assistance System (ADAS). The method extracts, from an input video stream including a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks. The perception tasks include object detection and other perception tasks. The method concurrently solves, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs. Each respective different branch corresponds to a respective one of the different perception tasks. The method forms a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs. The method controls an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.

## Claims

### Claim 1

A computer-implemented method in an Advanced Driver-Assistance System (ADAS), comprising:
extracting, by a hardware processor from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks;
concurrently solving, by the hardware processor using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks;
forming a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and
controlling an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the other perception tasks comprise semantic segmentation, depth estimation, and occlusion reasoning.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the hardware processor consists of a single GPU.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
associating bounding boxes across different pictures to obtain object tracks;
processing the bounding boxes to determine camera pose; and
localizing objects encapsulated by the bounding boxes responsive to the object tracks and the camera pose to provide refined object tracks for forming the at least one top-view map.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the refined object tracks are provided to be consistent over a given time period.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, further comprising generating a confidence score for each of the bounding boxes, and wherein the confidence score is used to obtain the object tracks.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the multi-task CNN comprises a plurality of sub-CNNs, each of the plurality of sub-CNNs for processing a respective different one of the different perception tasks.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprising training the multi-task CNN using training data comprising two-dimensional object boxes, three-dimensional object boxes, sparse three-dimensional points, and semantic pixels.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the training data is annotated for a respective of the different perception tasks.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein each of the semantic pixels are assigned to one of a plurality of available semantic categories.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising forming the top-view map using occlusion reasoning, the occlusion reasoning estimating semantics and distances for any pixels in the input video stream occluded by foreground objects.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein occluded areas of a scene in a frame of the input video stream are unannotated for the occlusion reasoning.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the collision avoidance comprises controlling a vehicle input selected from the group consisting of braking and steering.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, further comprising performing a scene understanding task responsive to the top-view map to aid in the collision avoidance.

### Claim 15

A computer program product for advanced driver-assistance, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
extracting, from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks;
concurrently solving, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks;
forming a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and
controlling an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.

### Claim 16 (depends on 15)

The computer program product of claim 15, wherein the other perception tasks comprise semantic segmentation, depth estimation, and occlusion reasoning.

### Claim 17 (depends on 15)

The computer program product of claim 15, wherein the hardware processor consists of a single GPU.

### Claim 18 (depends on 15)

The computer program product of claim 15, further comprising:
associating bounding boxes across different pictures to obtain object tracks;
processing the bounding boxes to determine camera pose; and
localizing objects encapsulated by the bounding boxes responsive to the object tracks and the camera pose to provide refined object tracks for forming the at least one top-view map.

### Claim 19 (depends on 15)

The computer program product of claim 15, wherein the multi-task CNN comprises a plurality of sub-CNNs, each of the plurality of sub-CNNs for processing a respective different one of the different perception tasks.

### Claim 20

A computer processing system for advanced driver-assistance, comprising:
a memory device including program code stored thereon;
a hardware processor, operatively coupled to the memory device, and configured to run the program code stored on the memory device to
extract, from an input video stream comprising a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks, the different perception tasks comprising object detection and other perception tasks;
concurrently solve, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs, each of the respective different branches corresponding to a respective one of the different perception tasks;
form a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs; and
control an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.
