Remote operation method, storage medium, remote operation device, and remote operation system

A non-transitory computer-readable storage medium storing a program, the program causing a remote operation computer to perform:

• converting an operation input to a joint angle command value of the robot when an operator performs an operation;
• acquiring actual visual information of a space in which the robot is present;
• generating virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;
• and
• projecting the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and projecting a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired.

A remote operation device comprising:

• a conversion unit configured to convert an operation input to a joint angle command value of the robot when an operator performs an operation;
• an acquisition unit configured to acquire actual visual information of a space in which the robot is present;
• a generation unit configured to generate virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;
• and
• a projection unit configured to project the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and to project a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired.

A remote operation system comprising a remote operation device and a remote-location device,

• wherein the remote-location device includes:
• a robot;
• and
• a visual sensor detecting an actual visual image of an environment in which the robot is present, and
• wherein the remote operation device includes:
• a conversion unit configured to convert an operation input to a joint angle command value of the robot when an operator performs an operation;
• an acquisition unit configured to acquire actual visual information of a space in which the robot is present;
• a generation unit configured to generate virtual visual information of an arbitrary viewpoint in the space in which the robot is present for the operator using a three-dimensional representation;
• and
• a projection unit configured to project the virtual visual information to a screen viewed by the operator when the actual visual information is not newly acquired and to project a mixed image in which an image based on the newly acquired actual visual information is combined with the virtual visual information when the actual visual information is newly acquired.