# US 12,008,817 B2 · Systems and methods for depth estimation in a vehicle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,817 B2
- Document ID: US12008817B2
- Status: Granted
- Kind: B2
- Application number: 17/198,954
- Filed: 2021-03-11
- Published: 2022-09-15
- Granted: 2024-06-11
- Earliest claimed: 2021-03-11
- Assignees: GM Global Technology Operations LLC
- Applicants: GM Global Technology Operations LLC
- Inventors: Albert Shalumov; Michael Slutsky
- Web page: https://patentgrep.com/patent/US12008817B2

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/096: Transfer learning
- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 3/4046: using neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 7/13: Edge detection
- G06T 7/55: from multiple images
- G06V 10/16: {using multiple overlapping images; Image stitching}
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/82: using neural networks
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- G06V 30/274: {Syntactic or semantic context, e.g. balancing}
- H04N 7/181: {for receiving images from a plurality of remote sources}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01S 5/16: using electromagnetic waves other than radio waves
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30261: Obstacle
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

Methods and system for training a neural network for depth estimation in a vehicle. The methods and systems receive respective training image data from at least two cameras. Fields of view of adjacent cameras of the at least two cameras partially overlap. The respective training image data is processed through a neural network providing depth data and semantic segmentation data as outputs. The neural network is trained based on a loss function. The loss function combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term. The panoramic loss term includes a similarity measure regarding overlapping image patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras. The semantic segmentation loss term quantifies a difference between ground truth semantic segmentation data and the semantic segmentation data output from the neural network.

## Claims

### Claim 1

A method of controlling a vehicle, the method comprising:
receiving, via at least one processor, respective image data from at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap and wherein the at least two cameras are mounted to the vehicle;
processing the respective image data through a neural network, wherein the neural network is trained to provide depth data and semantic segmentation data as outputs and wherein the neural network is trained using a loss function that combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term, wherein the panoramic loss term includes a similarity measure regarding overlapping patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras and wherein the panoramic loss term is calculated by steps including: projecting the respective image data into a combined equirectangular panoramic image; applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches; and performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches, wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth;
controlling, via the at least one processor, a function of the vehicle based on the depth data.

### Claim 2 (depends on 1)

The method of claim 1, wherein at least four cameras are mounted to the vehicle providing a surround view of the vehicle.

### Claim 3 (depends on 1)

The method of claim 1, wherein the plurality of loss terms includes a disparity loss term, wherein disparity is inverse of depth.

### Claim 4 (depends on 1)

The method of claim 1, wherein the disparity smoothness loss term is variable so as to be lower at image edges.

### Claim 5 (depends on 1)

The method of claim 1, comprising controlling, via the at least one processor, the function of the vehicle based on the depth data and the semantic segmentation data.

### Claim 6 (depends on 1)

The method of claim 1, comprising performing obstacle detection and avoidance processing based on the depth data and the semantic segmentation data and controlling at least one of steering, braking and propulsion of the vehicle based on the output of the obstacle detection and avoidance processing.

### Claim 7

A method of training a neural network and using the trained neural network to control a vehicle, the method comprising:
receiving, via at least one processor, respective training image data representing image data received from at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap;
processing the respective training image data through a neural network providing depth data and semantic segmentation data as outputs;
receiving, via the at least one processor, ground truth data including ground truth semantic segmentation data;
training, via the at least one processor, the neural network based on a loss function, wherein the loss function combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term;
wherein the panoramic loss term includes a similarity measure regarding overlapping image patches of the respective training image data that each correspond to a region of overlapping fields of view of the adjacent cameras, and wherein the panoramic loss term is calculated by steps including:
projecting the respective image data into a combined equirectangular panoramic image;
applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches; and
performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches;
wherein the semantic segmentation loss term quantifies a difference between the ground truth semantic segmentation data and the semantic segmentation data output from the neural network; and
wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth;
using the trained neural network to process image data received from at least two vehicle cameras mounted to the vehicle, wherein fields of view of adjacent vehicle cameras of the at least two vehicle cameras partially overlap, thereby providing live depth data and live semantic segmentation data as outputs;
controlling, via the at least one processor, a function of the vehicle based on the live depth data.

### Claim 8 (depends on 7)

The method of claim 7, wherein at least four vehicle cameras are mounted to the vehicle providing a surround view of the vehicle.

### Claim 9 (depends on 7)

The method of claim 7, wherein the ground truth data includes ground truth disparity data, and wherein the disparity smoothness loss term quantifies a difference between the ground truth disparity data and disparity data derived from the depth data output from the neural network.

### Claim 10 (depends on 7)

The method of claim 7, wherein the disparity smoothness term quantifies a smoothness of disparity data that is derived from the depth data output from the neural network.

### Claim 11 (depends on 10)

The method of claim 10, wherein the disparity smoothness loss term is variable so as to have lower values at image edges.

### Claim 12 (depends on 10)

The method of claim 10, wherein the disparity smoothness loss term is calculated by steps including:
finding image edges; and
applying a smoothness control function that varies based on the found image edges.

### Claim 13 (depends on 12)

The method of claim 12, wherein the image edges are determined based on evaluating local image intensity changes.

### Claim 14 (depends on 7)

The method of claim 7, comprising controlling, via the at least one processor, the function of the vehicle based on the depth data and the semantic segmentation data.

### Claim 15

A vehicle, the vehicle comprising:
at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap and wherein the at least two cameras are mounted to the vehicle;
at least one processor in operable communication with the at least two cameras, the at least one processor configured to execute program instructions, wherein the program instructions are configured to cause the at least one processor to:
receive respective image data from the at least two cameras;
process the respective image data through a neural network, wherein the neural network is trained to provide depth data and semantic segmentation data as outputs and wherein the neural network is trained using a loss function that combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term, wherein the panoramic loss term includes a similarity measure regarding overlapping patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras, and wherein the panoramic loss term is calculated by steps including: projecting the respective image data into a combined equirectangular panoramic image;
applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches; and performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches, wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth; and
control a function of the vehicle based on the depth data.

### Claim 16 (depends on 15)

The vehicle of claim 15, wherein at least four cameras are mounted to the vehicle providing a surround view of the vehicle.

### Claim 17 (depends on 15)

The vehicle of claim 15, wherein the disparity smoothness loss term is variable so as to have lower values at image edges.