Robot remote operation control device, robot remote operation control system, robot remote operation control method, and non-transitory computer readable medium

A robot remote operation control method, wherein in a robot remote operation which recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot to grip an object, the robot remote operation control method performed by a processor comprises:

• acquiring an environment sensor value related to the robot or a surrounding environment of the robot;
• acquiring an operator sensor value which detects the movement of the operator from a head mounted display and a hand wear controller worn by the operator;
• estimating a motion of the operator, which is a motion instruction with respect to the robot, by using a line of sight and a hand movement of the operator based on a trained model from the operator sensor value;
• generating a control command for gripping the object based on an estimated result of the motion of the operator, the environment sensor value and the operator sensor value, and provide the control command to the robot,
• wherein estimating the motion of the operator further comprises:
• extracting a feature amount based on the operator sensor value and the environment sensor value and classifying an action of the operator based on the extracted feature amount;
• calculating a likelihood of an alignment between the object and an alignment reference object for each of a plurality of placement candidates to perform a placement of the object gripped by the robot by using the environment sensor value and the operator sensor value;
• performing an alignment support to perform the alignment during the placement by using a placement candidate of the plurality of placement candidates having a highest likelihood of the alignment between the object and the alignment reference object;
• and
• predicting in advance on whether the operator is to perform the alignment at a stage in which the operator holds the object in hand by using information acquired from the operator sensor value,
• wherein the alignment is an operation to align the gripped object with respect to the alignment reference object.

A non-transitory computer readable medium, storing a program, wherein in a robot remote operation which recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot to grip an object, the program causes a computer to:

• acquire an environment sensor value related to the robot or a surrounding environment of the robot;
• acquire an operator sensor value, which is information indicating the movement of the operator that is detected from a head mounted display and a hand wear controller worn by the operator;
• receive the environment sensor value and the operator sensor value to estimate a motion of the operator, which is a motion instruction with respect to the robot, by using a line of sight and a hand movement of the operator based on a trained model from the operator sensor value;
• generate a control command for gripping the object based on an estimated result, the environment sensor value and the operator sensor value, and provide the control command to the robot;
• wherein estimating the motion of the operator further comprises:
• extracting a feature amount based on the operator sensor value and the environment sensor value and classifying an action of the operator based on the extracted feature amount;
• calculating a likelihood of an alignment between the object and an alignment reference object for each of a plurality of placement candidates to perform a placement of the object gripped by the robot by using the environment sensor value and the operator sensor value;
• performing an alignment support to perform the alignment during the placement by using a placement candidate of the plurality of placement candidates having a highest likelihood of the alignment between the object and the alignment reference object;
• and
• predicting in advance on whether the operator is to perform the alignment at a stage in which the operator holds the object in hand by using information acquired from the operator sensor value,
• wherein the alignment is an operation to align the gripped object with respect to the alignment reference object.

A robot remote operation control device, wherein in a robot remote operation which recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot to grip an object, the robot remote operation control device comprises:

• a processor, configured to perform
• acquiring an environment sensor value acquired by an environment sensor provided in the robot or a surrounding environment of the robot and an operator sensor value detecting the movement of the operator from a head mounted display and a hand wear controller worn by the operator;
• receiving the environment sensor value and the operator sensor value to estimate an intention of the operator based on a probability distribution with respect to operation candidate objects calculated from a gaze point of the operator in a robot world and a trajectory of a hand of the operator in the robot world,
• wherein the processor is configured to perform receiving the environment sensor value and the operator sensor value to estimate the intention of the operator based on the probability distribution with respect to all the operation candidate objects calculated from the gaze point of the operator in the robot world and the trajectory of the hand of the operator in the robot world further includes:
• estimating the gaze point based on a sight line information of the operator acquired from the head mounted display, and estimate a location of the hand of the operator in the robot world based on the hand motion information of the operator acquired from the hand wear controller,
• calculating first distances between the gaze point and the operation candidate objects and second distances between the hand and the operation candidate objects, and
• calculating the probability distribution with respect to all the operation candidate objects based on the first distances between the gaze point and the objects as operation candidates and the second distances between the hand and the objects as operation candidates, wherein the probability distribution is determined based on numbers of times that the gaze point of the operator is overlapped with the objects as the operation candidates.

A robot remote operation control method, wherein in a robot remote operation which recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot to grip an object, the robot remote operation control method performed by a processor comprises:

• acquiring an environment sensor value acquired by an environment sensor provided in the robot or a surrounding environment of the robot and an operator sensor value detecting the movement of the operator from a head mounted display and a hand wear controller worn by the operator;
• receiving the environment sensor value and the operator sensor value to estimate an intention of the operator based on a probability distribution with respect to operation candidate objects calculated from a gaze point of the operator in a robot world and a trajectory of a hand of the operator in the robot world, wherein-the processor is configured to perform receiving the environment sensor value and the operator sensor value to estimate the intention of the operator based on the probability distribution with respect to all the operation candidate objects calculated from the gaze point of the operator in the robot world and the trajectory of the hand of the operator in the robot world further includes:
• estimating the gaze point based on a sight line information of the operator acquired from the head mounted display, and estimate a location of the hand of the operator in the robot world based on the hand motion information of the operator acquired from the hand wear controller,
• calculating first distances between the gaze point and the operation candidate objects and second distances between the hand and the operation candidate objects, and
• calculating the probability distribution with respect to all the operation candidate objects based on the first distances between the gaze point and the objects as operation candidates and the second distances between the hand and the objects as operation candidates, wherein the probability distribution is determined based on numbers of times that the gaze point of the operator is overlapped with the objects as the operation candidates.

A non-transitory computer readable medium, storing a program, wherein in a robot remote operation which recognizes a movement of an operator and transmits the movement of the operator to a robot to operate the robot to grip an object, the program causes a computer to:

• acquire an environment sensor value acquired by an environment sensor provided in the robot or a surrounding environment of the robot and an operator sensor value detecting the movement of the operator from a head mounted display and a hand wear controller worn by the operator;
• receiving the environment sensor value and the operator sensor value to estimate an intention of the operator based on a probability distribution with respect to operation candidate objects calculated from a gaze point of the operator in a robot world and a trajectory of a hand of the operator in the robot world, wherein the processor is configured to perform receiving the environment sensor value and the operator sensor value to estimate the intention of the operator based on the probability distribution with respect to all the operation candidate objects calculated from the gaze point of the operator in the robot world and the trajectory of the hand of the operator in the robot world further includes:
• estimating the gaze point based on a sight line information of the operator acquired from the head mounted display, and estimate a location of the hand of the operator in the robot world based on the hand motion information of the operator acquired from the hand wear controller,
• calculating first distances between the gaze point and the operation candidate objects and second distances between the hand and the operation candidate objects, and
• calculating the probability distribution with respect to all the operation candidate objects based on the first distances between the gaze point and the objects as operation candidates and the second distances between the hand and the objects as operation candidates, wherein the probability distribution is determined based on numbers of times that the gaze point of the operator is overlapped with the objects as the operation candidates.