# US 2023/0321812 A1 · Remote control system, control method for remote control system, and remote control program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0321812 A1
- Document ID: US20230321812A1
- Status: Published application
- Kind: A1
- Application number: 18/209,045
- Filed: 2023-06-13
- Published: 2023-10-12
- Earliest claimed: 2020-12-16
- Assignees: Kawasaki Jukogyo Kabushiki Kaisha
- Applicants: Kawasaki Jukogyo Kabushiki Kaisha
- Inventors: Kentaro AZUMA; Hitoshi HASUNUMA
- Web page: https://patentgrep.com/patent/US20230321812A1

## CPC

- B25J 3/00: Manipulators of leader-follower type, i.e. both controlling unit and controlled unit perform corresponding spatial movements
- B25J 9/1689: {Teleoperation}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/025: {comprising haptic means}
- B25J 13/085: {Force or torque sensors (B25J13/082, B25J13/084 take precedence)}
- B25J 19/023: {including video camera means}
- H04N 23/53: of electronic viewfinders, e.g. rotatable or detachable
- G05B 2219/39529: Force, torque sensor in wrist, end effector
- G05B 2219/40165: Sensor data to display depends on robot status
- G05B 2219/40169: Display of actual situation at the remote site
- G05B 2219/40609: Camera to monitor end effector as well as object to be handled
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

A remote control system includes: an operator that is operated by a user; a robot that applies a treatment to an object in accordance with an action of the operator; a contact force sensor that is disposed in the robot and detects an operating state of the robot; an imager that captures images of at least one of the robot or the object; a display that displays the captured images captured by the imager and presents the captured images to the user operating the operator; and a controller that performs action control of at least one of the robot or the operator in accordance with detection results of the contact force sensor. The controller delays the action control to reduce a lag between the action control and display timings of the captured images by the display.

## Claims

### Claim 1

A remote control system comprising:
a master that is operated by a user;
a slave that applies a treatment to an object in accordance with an action of the master;
a sensor that is disposed in the slave and detects an operating state of the slave;
an imager that captures images of at least one of the slave or the object;
a display that displays the captured images captured by the imager and presents the captured images to the user operating the master; and
a controller that performs action control of at least one of the master or the slave based on detection results of the sensor, wherein
the controller delays the action control to reduce a lag between the action control and display timings of the captured images by the display.

### Claim 2 (depends on 1)

The remote control system according to claim 1, wherein
the controller receives the detection results from the slave, receives the captured images from the display, and obtains a delay amount in delaying the action control based on a time lag between the detection results and the captured images.

### Claim 3 (depends on 2)

The remote control system according to claim 2, wherein
in a case where the controller updates the delay amount to a new delay amount obtained based on the time lag, the controller updates the delay amount to the new delay amount stepwise.

### Claim 4 (depends on 2)

The remote control system according to claim 2, wherein
association information is added to at least one of the detection result or the captured image acquired by the sensor and the imager, respectively, at corresponding timings, the association information indicating that the detection result and the captured image are associated with each other, and
the controller distinguishes the detection result and the captured image acquired at the corresponding timings based on the association information from the detection results and the captured images received by the controller, and obtains the time lag.

### Claim 5 (depends on 4)

The remote control system according to claim 4, further comprising
an image processor that receives the captured images from the imager, processes the captured images, and transmits the processed captured images to the display, wherein
the image processor receives the detection results from the slave, adds, as the association information, the detection result acquired at the timing corresponding to the captured image to the captured image, and transmits the captured image to which the detection result is added, to the display, and
the controller receives the captured image to which the detection result is added from the display, and obtains the time lag based on a comparison between the detection result added to the captured image and the detection result received from the slave.

### Claim 6 (depends on 5)

The remote control system according to claim 5, wherein
the image processor combines the captured image with the detection result acquired at the timing corresponding to the captured image as an image to thereby add the detection result to the captured image, encodes the captured image to which the detection result is added, and transmits the encoded captured image to the display, and
the controller acquires the detection results from the captured images decoded by the display.

### Claim 7 (depends on 1)

The remote control system according to claim 1, wherein
as the action control, the controller controls an action of the slave and controls an action of the master such that a reaction force exerted on the slave is presented to the user, in accordance with operation of the master by the user and the detection results.

### Claim 8 (depends on 7)

The remote control system according to claim 7, wherein
the sensor is a force sensor.

### Claim 9

A remote control system comprising:
a master that is operated by a user in order to move a slave that applies a treatment to an object;
a display that displays captured images captured by an imager that captures images of at least one of the slave or the object, and presents the captured images to the user operating the master; and
a controller that performs action control of at least one of the master or the slave based on detection results of a sensor that is disposed in the slave and detects an operating state of the slave, wherein
the controller delays the action control to reduce a lag between the action control and display timings of the captured images by the display.

### Claim 10

A control method for a remote control system including
a master that is operated by a user,
a slave that applies a treatment to an object in accordance with an action of the master,
a sensor that is disposed in the slave and detects an operating state of the slave,
an imager that captures images of at least one of the slave or the object, and
a display that displays the captured images captured by the imager and presents the captured images to the user operating the master, the method comprising:
performing action control of at least one of the master or the slave based on detection results of the sensor; and
delaying the action control to reduce a lag between the action control and display timings of the captured images by the display.

### Claim 11

A non-transitory recording medium recording a remote control program for causing a computer to execute the function of controlling a remote control system including
a master that is operated by a user,
a slave that applies a treatment to an object in accordance with an action of the master,
a sensor that is disposed in the slave and detects an operating state of the slave,
an imager that captures images of at least one of the slave or the object, and
a display that displays the captured images captured by the imager and presents the captured images to the user operating the master, the program causing the computer to execute the functions of:
performing action control of at least one of the master or the slave based on detection results of the sensor; and
delaying the action control to reduce a lag between the action control and display timings of the captured images by the display.
