Method and apparatus for training neural network for imitating demonstrator's behavior

A method for training a Neural Network (NN) model for imitating behavior of a demonstrator, comprising the following steps:

• obtaining demonstration data representing a behavior of the demonstrator for performing a task, the demonstration data includes state data, action data and option data, wherein the state data of the demonstration data correspond to a condition for performing the task, the option data of the demonstration data correspond to subtasks of the task, and the action data of the demonstration data correspond to actions of the demonstrator performed for the task;
• sampling learner data representing a behavior of the NN model for performing the task based on a current learned policy, the learner data includes state data, action data and option data, wherein the state data of the learner data correspond to a condition for performing the task, the option data of the learner data correspond to subtasks of the task, and the action data of the learner data correspond to actions of NN model performed for the task, the policy includes a high level policy part for determining a current option and a low level policy part for determining a current action;
• and
• updating the policy by using a generative adversarial imitation learning (GAIL) process based on the demonstration data and the learner data.

A method for training a Neural Network (NN) model for self-driving assistance, comprising the following steps:

• training the NN model for self-driving assistance by:
• obtaining demonstration data representing a behavior of a demonstrator for performing a task, the demonstration data includes state data, action data and option data, wherein the state data of the demonstration data correspond to a condition for performing the task, the option data of the demonstration data correspond to subtasks of the task, and the action data of the demonstration data correspond to actions of the demonstrator performed for the task,
• sampling learner data representing a behavior of the NN model for performing the task based on a current learned policy, the learner data includes state data, action data and option data, wherein the state data of the learner data correspond to a condition for performing the task, the option data of the learner data correspond to subtasks of the task, and the action data of the learner data correspond to actions of NN model performed for the task, the policy includes a high level policy part for determining a current option and a low level policy part for determining a current action, and
• updating the policy by using a generative adversarial imitation learning (GAIL) process based on the demonstration data and the learner data;
• wherein the demonstration data represents a driver's behavior for driving a vehicle.

A method for training a Neural Network (NN) model for controlling robot locomotion, comprising:

• training the NN model for controlling robot locomotion by:
• obtaining demonstration data representing a behavior of a demonstrator for performing a task, the demonstration data includes state data, action data and option data, wherein the state data of the demonstration data correspond to a condition for performing the task, the option data of the demonstration data correspond to subtasks of the task, and the action data of the demonstration data correspond to actions of the demonstrator performed for the task,
• sampling learner data representing a behavior of the NN model for performing the task based on a current learned policy, the learner data includes state data, action data and option data, wherein the state data of the learner data correspond to a condition for performing the task, the option data of the learner data correspond to subtasks of the task, and the action data of the learner data correspond to actions of NN model performed for the task, the policy includes a high level policy part for determining a current option and a low level policy part for determining a current action, and
• updating the policy by using a generative adversarial imitation learning (GAIL) process based on the demonstration data and the learner data;
• wherein the demonstration data represents a demonstrator's locomotion for performing a task.

A method for controlling a machine with a trained Neural Network (NN) model, comprising the following steps:

• collecting environment data related to performing a task by the machine;
• obtaining state data and option data for a current time instant based at least in part on the environment data;
• inferring action data for the current time instant based on the state data and the option data for the current time instant with the trained NN model;
• and
• controlling an action of the machine based on the action data for the current time.

A vehicle capable of self-driving assistance, comprising:

• sensors configured to collect at least a part of environment data related to performing self-driving assistance by the vehicle;
• one or more processors;
• and
• one or more non-transitory storage devices storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps:
• collecting environment data related to performing a task by the vehicle,
• obtaining state data and option data for a current time instant based at least in part on the environment data,
• inferring action data for the current time instant based on the state data and the option data for the current time instant with the trained NN model, and
• controlling an action of the vehicle based on the action data for the current time.

A robot capable of automatic locomotion, comprising:

• sensors configured to collect at least a part of environment data related to performing automatic locomotion by the robot;
• one or more processors;
• and
• one or more non-transitory storage devices storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps:
• collecting environment data related to performing a task by the robot,
• obtaining state data and option data for a current time instant based at least in part on the environment data,
• inferring action data for the current time instant based on the state data and the option data for the current time instant with the trained NN model, and
• controlling an action of the robot based on the action data for the current time.

A computer system, comprising:

• one or more processors;
• and
• one or more non-transitory storage devices storing computer-executable instructions for training a Neural Network (NN) model for imitating behavior of a demonstrator, the instructions, when executed by the one or more processors, cause the one or more processors to perform:
• obtaining demonstration data representing a behavior of the demonstrator for performing a task, the demonstration data includes state data, action data and option data, wherein the state data of the demonstration data correspond to a condition for performing the task, the option data of the demonstration data correspond to subtasks of the task, and the action data of the demonstration data correspond to actions of the demonstrator performed for the task,
• sampling learner data representing a behavior of the NN model for performing the task based on a current learned policy, the learner data includes state data, action data and option data, wherein the state data of the learner data correspond to a condition for performing the task, the option data of the learner data correspond to subtasks of the task, and the action data of the learner data correspond to actions of NN model performed for the task, the policy includes a high level policy part for determining a current option and a low level policy part for determining a current action, and
• updating the policy by using a generative adversarial imitation learning (GAIL) process based on the demonstration data and the learner data.

One or more non-transitory computer readable storage media on which are stored computer-executable instructions for training a Neural Network (NN) model for imitating behavior of a demonstrator, the computer-executable instructions, when executed by one or more processors, cause the one or more processor to perform the following steps:

• obtaining demonstration data representing a behavior of the demonstrator for performing a task, the demonstration data includes state data, action data and option data, wherein the state data of the demonstration data correspond to a condition for performing the task, the option data of the demonstration data correspond to subtasks of the task, and the action data of the demonstration data correspond to actions of the demonstrator performed for the task;
• sampling learner data representing a behavior of the NN model for performing the task based on a current learned policy, the learner data includes state data, action data and option data, wherein the state data of the learner data correspond to a condition for performing the task, the option data of the learner data correspond to subtasks of the task, and the action data of the learner data correspond to actions of NN model performed for the task, the policy includes a high level policy part for determining a current option and a low level policy part for determining a current action;
• and
• updating the policy by using a generative adversarial imitation learning (GAIL) process based on the demonstration data and the learner data.