Training policy neural networks in simulation using scene synthesis machine learning models

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining a plurality of images of a scene in a real-world environment with which a robot will interact and, for each image, corresponding camera data comprising a viewpoint of a camera that captured the image;
• training a scene synthesis machine learning model using the plurality of images and the corresponding camera data, wherein the scene synthesis machine learning model is configured to receive a scene input that comprises a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint;
• and
• generating, using at least synthetic images generated by the scene synthesis machine learning model, training data for training a policy neural network for use in controlling the robot in the real-world environment to perform one or more tasks, wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of the environment and to generate as output a policy output defining an action to be performed by the robot in response to the observation, wherein the observation comprises an image of the environment captured by a robot camera of the robot, and wherein generating the training data comprises:
• generating, from synthetic images generated by the scene synthesis machine learning model, observations of scenes in a simulation of the environment being interacted with by a model of the robot.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining a plurality of images of a scene in a real-world environment with which a robot will interact and, for each image, corresponding camera data comprising a viewpoint of a camera that captured the image;
• training a scene synthesis machine learning model using the plurality of images and the corresponding camera data, wherein the scene synthesis machine learning model is configured to receive a scene input that comprises a camera viewpoint and to generate as output a synthetic image of the scene from the camera viewpoint;
• and
• generating, using at least synthetic images generated by the scene synthesis machine learning model, training data for training a policy neural network for use in controlling the robot in the real-world environment to perform one or more tasks, wherein the policy neural network is configured to receive a policy input comprising an observation characterizing a current state of the environment and to generate as output a policy output defining an action to be performed by the robot in response to the observation, wherein the observation comprises an image of the environment captured by a robot camera of the robot, and wherein generating the training data comprises:
• generating, from synthetic images generated by the scene synthesis machine learning model, observations of scenes in a simulation of the environment being interacted with by a model of the robot.