Remote control system, remote control method, and control program

A remote control method by a remote control system configured to remotely control a device to be operated comprising an end effector, the remote control method comprising:

• shooting an image of a first predetermined area of an environment in which the device to be operated is located;
• recognizing, based on the shot image of the first predetermined area, objects that can be grasped by the end effector;
• receiving, in an operation terminal that displays the shot image, handwritten information input to the displayed shot image,
• shooting, when a part of an area of the shot image designated by the handwritten information is a predetermined outer peripheral area of the shot image displayed on the operation terminal before the input of the handwritten information is completed, an image of a second predetermined area including the part of the area of the shot image and an area adjacent to the first predetermined area, and causing the shot image to be displayed on the operation terminal;
• and
• estimating, based on the objects that can be grasped and the handwritten information, an object to be grasped which has been requested to be grasped by the end effector from among the objects that can be grasped and estimating a content of a motion with regard to the object to be grasped which is requested to be performed by the device to be operated.

A non-transitory computer readable medium storing a control program for causing a computer to execute remote control processing by a remote control system configured to remotely control a device to be operated comprising an end effector, the control program further causing a computer to perform the processing of:

• shooting an image of a first predetermined area of an environment in which the device to be operated is located;
• recognizing, based on the shot image of the first predetermined area, objects that can be grasped by the end effector;
• receiving, in an operation terminal that displays the shot image, handwritten information input to the displayed shot image,
• shooting, when a part of an area of the shot image designated by the handwritten information is a predetermined outer peripheral area of the shot image displayed on the operation terminal before the input of the handwritten information is completed, an image of a second predetermined area including the part of the area of the shot image and an area adjacent to the first predetermined area, and causing the shot image to be displayed on the operation terminal;
• and
• estimating, based on the objects that can be grasped and the handwritten information, an object to be grasped which has been requested to be grasped by the end effector from among the objects that can be grasped and estimating a content of a motion with regard to the object to be grasped which is requested to be performed by the device to be operated.