Determining structure and motion in images using neural networks

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:

• a motion neural network configured to:
• receive a first image and a second image, and
• process the first image and the second image to generate (i) a motion output characterizing motion between the first image and the second image, and (ii) a motion segmentation mask output;
• and
• a scene structure neural network configured to:
• receive the first image, and
• process the first image to generate a depth map characterizing, for each of the plurality of pixels in the first image, a depth of the scene at the pixel;
• and
• a subsystem configured to generate, from the depth map, the motion output, and the motion segmentation mask output, an optical flow output that characterizes motion of pixels in the first image from the first image to the second image.