# US 2021/0200202 A1 · Remote control device and method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0200202 A1
- Document ID: US20210200202A1
- Status: Published application
- Kind: A1
- Application number: 16/842,161
- Filed: 2020-04-07
- Published: 2021-07-01
- Earliest claimed: 2019-12-30
- Assignees: Hanwha Defense Co., Ltd.
- Applicants: Hanwha Defense Co., Ltd.
- Inventors: Hee Seo CHAE
- Web page: https://patentgrep.com/patent/US20210200202A1

## CPC

- G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)}
- G05D 1/0027: {involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00)}
- G05D 1/0038: {by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A remote control device includes: a display configured to receive a driving image and a monitoring image from each of N mobile robots, such that N driving images and M monitoring images are received, and to display the received N driving images and M monitoring images; a sensor configured to track a gaze of a user; and a controller configured to provide a control right to the user for a first robot of the N mobile robots as a master mobile robot based on the gaze of the user being tracked by the sensor as staying on the driving image or the monitoring image of the first robot displayed on the display.

## Claims

### Claim 1

A remote control device comprising:
a display configured to receive a driving image and a monitoring image from each of N mobile robots, such that N driving images and M monitoring images are received, and to display the received N driving images and M monitoring images;
a sensor configured to track a gaze of a user; and
a controller comprising at least one processor, the controller configured to:
provide a control right to the user for a first robot of the N mobile robots as a master mobile robot based on the gaze of the user being tracked by the sensor as staying on the driving image or the monitoring image of the first robot displayed on the display; and
based on the user selecting, via the display, one of the N driving images or one of the M monitoring images, other than the driving image and the monitoring image of the first robot that is the master mobile robot, within a certain time after the user is provided the control right of the master mobile robot, perform collective control of the master mobile robot and another mobile robot from which the selected driving image or the selected monitoring image was received.

### Claim 2 (depends on 1)

The remote control device of claim 1, wherein the display is configured to:
when the gaze of the user is tracked by the sensor as staying on the driving image of the first robot displayed on the display, provide an interface to the user to select one of the N driving images, other than the driving image from the first robot that is the master mobile robot.

### Claim 3 (depends on 1)

The remote control device of claim 1, wherein the display is configured to:
when the gaze of the user is tracked by the sensor as staying on the monitoring image of the first robot displayed on the display, provide an interface to the user to select one of the M monitoring images, other than the monitoring image from the first robot that is the master mobile robot.

### Claim 4 (depends on 2)

The remote control device of claim 2, wherein the controller is configured to perform the collective control of the master mobile robot and the another mobile robot, corresponding to the driving image selected via the interface, such that the master mobile robot and the another mobile robot move to a point corresponding to where the gaze of the user is tracked by the sensor moves on the driving image of the master mobile robot that is displayed on the display.

### Claim 5 (depends on 2)

The remote control device of claim 2, wherein the controller is configured to collectively control driving of the master mobile robot and the another mobile robot, corresponding to the driving image selected via the interface, based on an input from a joystick.

### Claim 6 (depends on 3)

The remote control device of claim 3, wherein the controller is configured to perform the collective control of the master mobile robot and the another mobile robot, corresponding to the monitoring image selected via the interface, such that the master mobile robot and the another mobile robot move to a point corresponding to where the gaze of the user is tracked by the sensor moves on the monitoring image of the master mobile robot that is displayed on the display.

### Claim 7 (depends on 3)

The remote control device of claim 3, wherein the controller enables the master mobile robot and the another mobile robot, corresponding to the monitoring image selected via the interface, to perform shooting in a collective control method using a joystick.

### Claim 8 (depends on 1)

The remote control device of claim 1, the controller is configured to provide the control right to the user for the first robot as the master mobile robot at a time t1 based on the gaze of the user being tracked by the sensor as staying on a first image that is the driving image or the monitoring image of the first robot displayed on the display, and when the gaze of the user tracked by the sensor at a time t2 moves and stays for more than a preset time on a second image that is the driving image or the monitoring image of a second robot of the N mobile robots displayed on the display, set the second robot as the master mobile robot.

### Claim 9 (depends on 8)

The remote control device of claim 8, wherein the controller is configured to:
revoke the control right of the first robot, at the time t2, based on one of the first image of the first robot and the second image of the second robot being a driving image and the other of the first image of the first robot and the second image of the second robot being a monitoring image, and
maintain collective control rights of the first robot and the second robot, at the time t2, based both of the first image of the first robot and the second image of the second robot being a same type from among a driving image and a monitoring image.

### Claim 10 (depends on 1)

The remote control device of claim 1, wherein the controller causes the display to display identification information indicating each of the N driving images and each of the M monitoring images displayed on the display that correspond to the master mobile robot or the another mobile robot of which the controller performs the collective control.

### Claim 11

A remote control method comprising:
receiving a driving image and a monitoring image from each of N mobile robots, such that N driving images and M monitoring images are received;
displaying, on a display, the received N driving images and M monitoring images;
tracking, via a sensor, a gaze of a user;
providing a control right to the user for a first robot of the N mobile robots as a master mobile robot based on the gaze of the user being tracked by the sensor as staying on the driving image or the monitoring image of the first robot displayed on the display; and
performing, based on the user selecting, via the display, one of the N driving images or one of the M monitoring images, other than the driving image and the monitoring image of the first robot that is the master mobile robot, within a certain time after the user is provided the control right of the master mobile robot, collective control of the master mobile robot and another mobile robot from which the selected driving image or the selected monitoring image was received.

### Claim 12 (depends on 11)

The remote control method of claim 11, further comprising:
activating, in a case where the gaze of the user is tracked by the sensor as staying on the driving image of the first robot displayed on the display, a first interface to the user to select one of the N driving images, other than the driving image from the first robot that is the master mobile robot; and
activating, in a case where the gaze of the user is tracked by the sensor as staying on the monitoring image of the first robot displayed on the display, a second interface to the user to select one of the M monitoring images, other than the monitoring image from the first robot that is the master mobile robot.

### Claim 13 (depends on 12)

The remote control method of claim 12, further comprising performing collective control of the master mobile robot and the another mobile robot, corresponding to the driving image selected via the first interface, such that the master mobile robot and the another mobile robot move to a point corresponding to where the gaze of the user is tracked by the sensor moves on the driving image of the master mobile robot that is displayed on the display.

### Claim 14 (depends on 12)

The remote control method of claim 12, further comprising performing collective control of the master mobile robot and the another mobile robot, corresponding to the monitoring image selected via the second interface, such that the master mobile robot and the another mobile robot shoot a point corresponding to where the gaze of the user is tracked by the sensor moves on the driving image of the master mobile robot that is displayed on the display.

### Claim 15 (depends on 11)

The remote control method of claim 11, wherein
the providing the control right to the user for the first robot as the master mobile robot is at a time t1, and the remote control method further comprises:
setting, in a case where the gaze of the user tracked by the sensor moves and stays at a time t2 for more than a preset time on the driving image or the monitoring image of a second robot of the N mobile robots displayed on the display, the second robot as the master mobile robot.

### Claim 16 (depends on 1)

The remote control device of claim 1, wherein the controller is configured to determine the one of the N driving images or the one of the M monitoring images is selected by the user based on the user touching the display.

### Claim 17

A non-transitory computer readable medium comprising computer instructions that are configured to, when executed by at least one processor, cause the at least one processor to:
display, via a display, N driving images and M monitoring images that are received, the N driving images and M monitoring images including a driving image and a monitoring image from each of N mobile robots;
track, via a sensor, a gaze of a user;
provide a control right to the user for a first robot of the N mobile robots as a master mobile robot based on the gaze of the user being tracked by the sensor as staying on the driving image or the monitoring image of the first robot displayed on the display; and
perform, based on the user selecting, via the display, one of the N driving images or one of the M monitoring images, other than the driving image and the monitoring image of the first robot that is the master mobile robot, within a certain time after the user is provided the control right of the master mobile robot, collective control of the master mobile robot and another mobile robot from which the selected driving image or the selected monitoring image was received.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the computer instructions are further configured to, when executed by the at least one processor, cause the at least one processor to:
activate, in a case where the gaze of the user is tracked by the sensor as staying on the driving image of the first robot displayed on the display, a first interface to the user to select one of the N driving images, other than the driving image from the first robot that is the master mobile robot; and
activate, in a case where the gaze of the user is tracked by the sensor as staying on the monitoring image of the first robot displayed on the display, a second interface to the user to select one of the M monitoring images, other than the monitoring image from the first robot that is the master mobile robot.

### Claim 19 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the computer instructions are further configured to, when executed by the at least one processor, cause the at least one processor to:
perform the collective control of the master mobile robot and the another mobile robot, corresponding to the driving image selected via the first interface, such that the master mobile robot and the another mobile robot move to a point corresponding to where the gaze of the user is tracked by the sensor moves on the driving image of the master mobile robot that is displayed on the display.

### Claim 20 (depends on 18)

The non-transitory computer readable medium of claim 18, wherein the computer instructions are further configured to, when executed by the at least one processor, cause the at least one processor to:
perform the collective control of the master mobile robot and the another mobile robot, corresponding to the monitoring image selected via the second interface, such that the master mobile robot and the another mobile robot shoot a point corresponding to where the gaze of the user is tracked by the sensor moves on the driving image of the master mobile robot that is displayed on the display.
