Robotic demonstration learning

A system comprising:

• one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• receiving perceptual data of a workcell of a robot configured to execute a task according to a skill template, wherein the skill template specifies a plurality of subtasks required to perform a skill, wherein the plurality of subtasks includes at least one demonstration subtask that relies on visual characteristics of the workcell;
• generating actual local demonstration data from a workcell demonstration of the demonstration subtask by operating the robot in the workcell while capturing sensor data generated by a plurality of sensors in the workcell of the robot, wherein the actual local demonstration data comprises task state representations, and each task state representation is generated by processing sensor output of each sensor of the plurality of sensors by a different respective neural network of a plurality of neural networks to generate, by each neural network, a different portion of the task state representation;
• obtaining a base control policy for the demonstration subtask;
• and
• tuning, using the actual local demonstration data from the workcell demonstration, the base control policy of the robot to follow an updated control policy for the demonstration subtask.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• receiving perceptual data of a workcell of a robot configured to execute a task according to a skill template, wherein the skill template specifies a plurality of subtasks required to perform a skill, wherein the plurality of subtasks includes at least one demonstration subtask that relies on visual characteristics of the workcell;
• generating actual local demonstration data from a workcell demonstration of the demonstration subtask by operating the robot in the workcell while capturing sensor data generated by a plurality of sensors in the workcell of the robot, wherein the actual local demonstration data comprises task state representations, and each task state representation is generated by processing sensor output of each sensor of the plurality of sensors by a different respective neural network of a plurality of neural networks to generate, by each neural network, a different portion of the task state representation;
• obtaining a base control policy for the demonstration subtask;
• and
• tuning, using the actual local demonstration data from the workcell demonstration, the base control policy of the robot to follow an updated control policy for the demonstration subtask.