Self-supervised training of a depth estimation model using depth hints

A computer-implemented method for training a depth estimation model comprising:

• accessing training image data comprising a plurality of image pairs, each image pair comprising a first image and a second image;
• for each image pair:
• generating, by the depth estimation model, depth prediction values for pixels of the first image;
• obtaining depth hint values for the pixels of the first image;
• projecting the second image onto the first image based on the depth prediction values for the pixels of the first image to generate a model synthetic frame;
• projecting the second image onto the first image based on the depth hint values for the pixels of the first image to generate a hinted synthetic frame;
• calculating primary loss values for the depth prediction values based on a comparison of the pixels of the model synthetic frame and the pixels of the first image;
• calculating hinted loss values for the depth hint values based on a comparison of the pixels of the hinted synthetic frame and the pixels of the first image;
• and
• calculating an overall loss for the image pair, the contribution of a given pixel in the first image to the overall loss determined based on whether the primary loss value or hinted loss value is smaller, wherein the contribution of a first pixel in the first image with a hinted loss value smaller than a primary loss value comprises the primary loss value and a supervised depth loss value based on a depth prediction value and a depth hint value for the first pixel;
• adjusting the parameters of the depth estimation model based on the overall losses of the image pairs.

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

• receiving an image of a scene;
• inputting the image into a depth estimation model that was trained by a process comprising:
• accessing training image data comprising a plurality of image pairs, each image pair comprising a first image and a second image;
• for each image pair:
• generating, by the depth estimation model, depth prediction values for pixels of the first image;
• obtaining depth hint values for the pixels of the first image;
• projecting the second image onto the first image based on the depth prediction values for the pixels of the first image to generate a model synthetic frame;
• projecting the second image onto the first image based on the depth hint values for the pixels of the first image to generate a hinted synthetic frame;
• calculating primary loss values for the depth prediction values based on a comparison of the pixels of the model synthetic frame and the pixels of the first image;
• calculating hinted loss values for the depth hint values based on a comparison of the pixels of the hinted synthetic frame and the pixels of the first image;
• and
• calculating an overall loss for the image pair, the contribution of a given pixel in the first image to the overall loss determined based on whether the primary loss value or hinted loss value is smaller, wherein the contribution of a first pixel in the first image with a hinted loss value smaller than a primary loss value comprises the primary loss value and a supervised depth loss value based on a depth prediction value and a depth hint value for the first pixel;
• adjusting the parameters of the depth estimation model based on the overall losses of the image pairs;
• and
• generating, by the depth estimation model, a depth map of the scene corresponding to the image of the scene.