Teleoperation assist device, teleoperation assist method, and storage medium

A teleoperation assist method that is performed by a teleoperation assist device remotely operating at least an end effector representative of a human hand, the teleoperation assist method comprising:

• acquiring information on a motion of an operator who operates the teleoperation assist device to cause at least the end effector to move, via a moving means, wherein the information on the motion of the operator includes information of a position of a finger of the operator and a movement of the finger of the operator;
• acquiring environment information of with respect to a first environment in which the end effector is operated and a second environment in which the teleoperation assist device is operated;
• estimating a target object which is a target associated with what the operator intends to operate by the end effector and a task which is an operation method of the target object related to how the operator intends to operate, by inputting the information associated with the motion of the operator and the environment information to a first model, the target object and the operator method changes according to the environment information;
• setting a first parameter associated with an operation type of the end effector and a second parameter associated with an operation of the moving means by inputting an estimate of the target object and the task which is the operation method of the target object to a second model;
• when a same object as the target object is not included in the second model, set the first parameter and the second parameter based on information of a similar object being similar to the target object in the second model;
• determining an amount associated with an operation of the end effector based on a grasp taxonomy of human grasp type representative of the first parameter and the information associated with the motion of the operator;
• determining an amount of operation associated with the moving means based on the second parameter, and the information associated with the motion of the operator.

A non-transitory computer-readable storage medium storing a program causing a computer of a teleoperation assist device remotely operating at least an end effector representative of a human hand to perform:

• acquiring information on a motion of an operator who operates the teleoperation assist device to cause, via a moving means, at least the end effector and the moving means to move, wherein the information on the motion of the operator includes information of a position of a finger of the operator and a movement of the finger of the operator;
• acquiring environment information of with respect to a first environment in which the end effector is operated and a second environment in which the teleoperation assist device is operated;
• estimating a target object which is a target associated with what the operator intends to operate by the end effector and a task which is an operation method of the target object related to how the operator intends to operate, by inputting information associated with the motion of the operator and the environment information to a first model, the target object and the operator method changes according to the environment information;
• setting a first parameter associated with an operation type of the end effector and a second parameter associated with an operation of the moving means by inputting an estimate of the target object and the task which is the operation method of the target object to a second model;
• when a same object as the target object is not included in the second model, set the first parameter and the second parameter based on information of a similar object being similar to the target object in the second model;
• determining an amount associated with an operation of the end effector based on a grasp taxonomy of human grasp type representative of the first parameter and the information associated with the motion of the operator;
• determining an amount of operation associated with the moving means based on the second parameter, and the information associated with the motion of the operator.