Self-supervised attention learning for depth and motion estimation

A method, comprising:

• receiving a first image from a first time from a camera;
• based on the first image, generating a depth map including depths between the camera and objects in the first image;
• generating a first pose of the camera based on the first image;
• generating a second pose of the camera for a second time based on a second image received from the camera before the first image;
• and
• generating a third pose of the camera for a third time based on a third image received from the camera after the first image;
• determining a first motion of the camera between the second time and the first time based on the first pose and the second pose;
• determining a second motion of the camera between the second time and the third time based on the second pose and the third pose;
• based on the first image, generate an attention map including attention coefficients indicative of amounts of attention to attribute to the objects in the first image;
• reconstructing the second image using the attention map, the depth map, and the first motion between the second time and the first time to produce a reconstructed second image;
• reconstructing the third image using the attention map, the depth map, and the second motion between the second time and the third time to produce a reconstructed third image;
• and
• based on at least one of the reconstructed second image and the reconstructed third image, selectively adjust at least one parameter.

A system, comprising:

• one or more processors;
• and
• memory including code that, when executed by the one or more processors, perform functions including:
• receiving a first image from a first time from a camera;
• based on the first image, generating a depth map including depths between the camera and objects in the first image;
• generating a first pose of the camera based on the first image;
• generating a second pose of the camera for a second time based on a second image received from the camera before the first image;
• and
• generating a third pose of the camera for a third time based on a third image received from the camera after the first image;
• determining a first motion of the camera between the second time and the first time based on the first pose and the second pose;
• determining a second motion of the camera between the second time and the third time based on the second pose and the third pose,
• based on the first image, generating an attention map including attention coefficients indicative of amounts of attention to attribute to the objects in the first image;
• reconstructing the second image using the attention map, the depth map, and the first motion between the second time and the first time to produce a reconstructed second image;
• reconstructing the third image using the attention map, the depth map, and the second motion between the second time and the third time to produce a reconstructed third image;
• and
• based on at least one of the reconstructed second image and the reconstructed third image, selectively adjusting at least one parameter.

A system, comprising:

• a first means for:
• receiving a first image from a first time from a camera;
• and
• based on the first image, generating a depth map including depths between the camera and objects in the first image;
• a second means for:
• generating a first pose of the camera based on the first image;
• generating a second pose of the camera for a second time based on a second image received from the camera before the first image;
• and
• generating a third pose of the camera for a third time based on a third image received from the camera after the first image;
• a third means for:
• determining a first motion of the camera between the second time and the first time based on the first pose and the second pose;
• and
• determining a second motion of the camera between the second time and the third time based on the second pose and the third pose;
• wherein the first means includes attention mechanisms configured to, based on the first image, generate an attention map including attention coefficients indicative of amounts of attention to attribute to the objects in the first image;
• a fourth means for reconstructing for reconstructing the second image using the attention map, the depth map, and the first motion between the second time and the first time to produce a reconstructed second image;
• a fifth means for reconstructing the third image using the attention map, the depth map, and the second motion between the second time and the third time to produce a reconstructed third image;
• and
• a sixth means for, based on at least one of the reconstructed second image and the reconstructed third image, selectively adjusting at least one parameter of at least one of the first means, the second means, and the third means.