Machine learning methods and apparatus related to predicting motion(s) of object(s) in a robot's environment based on image(s) capturing the object(s) and based on parameter(s) for future robot movement in the environment

A method implemented by one or more processors, comprising:

• generating candidate robot movement parameters, the candidate robot movement parameters defining at least a portion of a candidate movement performable in an environment of a robot by one or more components of the robot;
• identifying a current image captured by a vision sensor associated with the robot, the current image capturing at least a portion of the environment of the robot;
• applying the current image and the candidate robot movement parameters as input to a trained neural network employing an action-conditioned motion prediction model that explicitly models pixel motion by predicting a distribution over the pixel motion from previous frames;
• generating a plurality of predicted transformations of the current image, the predicted transformations each indicating respective pixel motion predictions and being generated based on the application of the current image and the candidate robot movement parameters to the trained neural network;
• transforming the current image, based on applying the predicted transformations to the current image, to generate a plurality of predicted images;
• compositing the plurality of predicted images to generate a predicted image comprising a plurality of pixels, the predicted image predicting the portion of the environment of the robot were the at least the portion of candidate movement to be performed in the environment by the components of the robot;
• determining, based on the predicted image, whether to perform the candidate movement;
• and
• responsive to determining, based on the predicted image, to perform the candidate movement:
• providing one or more control commands to one or more actuators of the robot to perform the candidate movement.