Integrating sensor streams for robotic demonstration learning

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• selecting, by a learning system for a robot, a base update rate for combining multiple sensor streams into a task state representation representing a state of a subtask being performed by the robot, wherein selecting the base update rate comprises selecting the base update rate based on a minimum update rate of the robot;
• generating, by each neural network of a plurality of neural networks, a respective portion of the task state representation at a respective update rate of a respective sensor of a plurality of sensors, wherein the plurality of sensors include one or more robot state sensors that generate sensor data representing physical characteristics of a robot and one or more perceptual sensors that generate sensor data representing visual characteristics of a workcell of the robot;
• repeatedly generating, by the learning system, the task state representation at the base update rate, including combining, during each time period defined by the update rate, the task state representation from most recently updated sensor data processed by the plurality of neural networks;
• and
• repeatedly using the task state representations to generate commands for the robot at the base update rate.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• selecting, by a learning system for a robot, a base update rate for combining multiple sensor streams into a task state representation representing a state of a subtask being performed by the robot, wherein selecting the base update rate comprises selecting the base update rate based on a minimum update rate of the robot;
• generating, by each neural network of a plurality of neural networks, a respective portion of the task state representation at a respective update rate of a respective sensor of a plurality of sensors, wherein the plurality of sensors include one or more robot state sensors that generate sensor data representing physical characteristics of a robot and one or more perceptual sensors that generate sensor data representing visual characteristics of a workcell of the robot;
• repeatedly generating, by the learning system, the task state representation at the base update rate, including combining, during each time period defined by the update rate, the task state representation from most recently updated sensor data processed by the plurality of neural networks;
• and
• repeatedly using the task state representations to generate commands for the robot at the base update rate.