Remote control system and remote control method

A remote control method performed by a remote control system configured to remotely control a device to be operated comprising an end effector, the remote control method comprising:

• shooting an environment in which the device to be operated is located;
• receiving, by an operation terminal displaying a shot image of the environment, handwritten input information input to the displayed shot image;
• recognizing objects that can be grasped by the end effector based on the shot image;
• based on the objects that can be grasped and the handwritten input information input to the shot image, estimating an object to be grasped which has been requested to be grasped by the end effector from among the objects that can be grasped and estimating a way of performing a grasping motion by the end effector, the grasping motion having been requested to be performed with regard to the object to be grasped;
• estimating a level of the grasping motion which has been requested to be performed with regard to the object to be grasped based on the objects that can be grasped which the recognition unit has recognized and the handwritten input information input to the shot image;
• and
• controlling the end effector to perform the grasping motion using a trajectory generated based upon the estimating,
• wherein the object to be grasped is a knob for a drawer or a door knob of a door, and when the object to be grasped is the knob for the drawer, the grasping motion is a motion for holding the knob and then opening and closing the drawer, and when the object to be grasped is the door knob of the door, the grasping motion is a motion for holding the door knob and then opening and closing the door, and
• wherein the level of the grasping motion indicates a degree of opening of the drawer or the door.

A non-transitory computer readable medium storing a program for causing a computer to:

• recognize, based on a shot image of an environment in which a device to be operated comprising an end effector is located, objects that can be grasped by the end effector;
• based on handwritten input information input to the shot image displayed on an operation terminal and the objects that can be grasped, estimate an object to be grasped which has been requested to be grasped by the end effector from among the objects that can be grasped and estimate a way of performing a grasping motion by the end effector, the grasping motion having been requested to be performed with regard to the object to be grasped;
• estimate a level of the grasping motion which has been requested to be performed with regard to the object to be grasped based on the objects that can be grasped which the recognition unit has recognized and the handwritten input information input to the shot image;
• and
• control the end effector to perform the grasping motion using a trajectory generated based upon the estimating,
• wherein the object to be grasped is a knob for a drawer or a door knob of a door, and when the object to be grasped is the knob for the drawer, the grasping motion is a motion for holding the knob and then opening and closing the drawer, and when the object to be grasped is the door knob of the door, the grasping motion is a motion for holding the door knob and then opening and closing the door, and
• wherein the level of the grasping motion indicates a degree of opening of the drawer or the door.