Systems and methods for depth estimation in a vehicle

A method of training a neural network and using the trained neural network to control a vehicle, the method comprising:

• receiving, via at least one processor, respective training image data representing image data received from at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap;
• processing the respective training image data through a neural network providing depth data and semantic segmentation data as outputs;
• receiving, via the at least one processor, ground truth data including ground truth semantic segmentation data;
• training, via the at least one processor, the neural network based on a loss function, wherein the loss function combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term;
• wherein the panoramic loss term includes a similarity measure regarding overlapping image patches of the respective training image data that each correspond to a region of overlapping fields of view of the adjacent cameras, and wherein the panoramic loss term is calculated by steps including:
• projecting the respective image data into a combined equirectangular panoramic image;
• applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches;
• and
• performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches;
• wherein the semantic segmentation loss term quantifies a difference between the ground truth semantic segmentation data and the semantic segmentation data output from the neural network;
• and
• wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth;
• using the trained neural network to process image data received from at least two vehicle cameras mounted to the vehicle, wherein fields of view of adjacent vehicle cameras of the at least two vehicle cameras partially overlap, thereby providing live depth data and live semantic segmentation data as outputs;
• controlling, via the at least one processor, a function of the vehicle based on the live depth data.

A vehicle, the vehicle comprising:

• at least two cameras, wherein fields of view of adjacent cameras of the at least two cameras partially overlap and wherein the at least two cameras are mounted to the vehicle;
• at least one processor in operable communication with the at least two cameras, the at least one processor configured to execute program instructions, wherein the program instructions are configured to cause the at least one processor to:
• receive respective image data from the at least two cameras;
• process the respective image data through a neural network, wherein the neural network is trained to provide depth data and semantic segmentation data as outputs and wherein the neural network is trained using a loss function that combines a plurality of loss terms including at least a semantic segmentation loss term and a panoramic loss term, wherein the panoramic loss term includes a similarity measure regarding overlapping patches of the respective image data that each correspond to a region of overlapping fields of view of the adjacent cameras, and wherein the panoramic loss term is calculated by steps including:
• projecting the respective image data into a combined equirectangular panoramic image;
• applying a mask to the combined equirectangular panoramic image so as to extract the overlapping image patches;
• and performing the similarity measure on the overlapping image patches to quantify the similarity of the overlapping image patches, wherein the plurality of loss terms further includes a disparity smoothness loss term, wherein disparity is inverse of depth;
• and
• control a function of the vehicle based on the depth data.