# US 2021/0078166 A1 · Display control device, display control method, computer program product, and communication system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0078166 A1
- Document ID: US20210078166A1
- Status: Published application
- Kind: A1
- Application number: 17/092,598
- Filed: 2020-11-09
- Published: 2021-03-18
- Earliest claimed: 2011-08-02
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Kenta KAWAMOTO; Yoshiaki IWAI; Satoru SHIMIZU
- Web page: https://patentgrep.com/patent/US20210078166A1

## CPC

- B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418)
- B25J 5/007: {mounted on wheels}
- B25J 9/0003: {Home robots, i.e. small robots for domestic use}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1697: {Vision controlled systems}
- B25J 13/06: Control stands, e.g. consoles, switchboards
- B25J 13/08: by means of sensing devices, e.g. viewing or touching devices
- B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)}
- G05B 2219/40099: Graphical user interface for robotics, visual robot user interface
- G05B 2219/40123: Indicate, select features on display, remote manipulator will execute
- G05B 2219/40409: Robot brings object near operator, operator places object in correct position
- G05B 2219/40411: Robot assists human in non-industrial environment like home or office
- Y10S 901/01: Mobile robot
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS

## Abstract

A control system, method and computer program product cooperate to assist control for an autonomous robot. An interface receives recognition information from an autonomous robot, said recognition information including candidate target objects to interact with the autonomous robot. A display control unit causes a display image to be displayed on a display of candidate target objects, wherein at least two of the candidate target objects are displayed with an associated indication of a target object score.

## Claims

### Claim at position 1

1: A movable apparatus comprising:
driving circuitry that controllably drives a movement of the movable apparatus;
a camera configured to capture a captured image and produce image data; and
circuitry configured to
generate current location information of the movable apparatus to a target position based on GPS,
transmit to an instructing device the current location information of the movable apparatus, and the image data, the instructing device displaying a map indicating a current location along a planned travel route of the movable apparatus based on the current location information and a visual indicator that indicates a candidate target object in the captured image based on the image data,
receive from the instructing device feedback information that indicates the candidate target object is correctly identified as the target object in the captured image, the feedback information relating to detected user gesture information that is received to control an operation of the movable apparatus, the visual indicator that indicates the target object being set based on the feedback information, and
control the movement of the movable apparatus based on the feedback information.

### Claim at position 2 (depends on 1)

2: The movable apparatus according to claim 1, wherein the feedback information is generated based on a command input indicating a selection of the target object.

### Claim at position 3 (depends on 1)

3: The movable apparatus according to claim 1, wherein the driving circuitry controls the movable apparatus to move towards the target object.

### Claim at position 4 (depends on 1)

4: The movable apparatus according to claim 1, wherein the circuitry is further configured to:
continuously obtain image data of the captured image, the captured image including the target object recognized based on the feedback information.

### Claim at position 5 (depends on 4)

5: The movable apparatus according to claim 4, wherein the circuitry is further configured to:
continuously transmit the image data of the captured image including the target object.

### Claim at position 6 (depends on 1)

6: The movable apparatus according to claim 1, wherein the circuitry is further configured to:
control the driving circuitry of the movable apparatus to move the movable apparatus.

### Claim at position 7 (depends on 1)

7: The movable apparatus according to claim 1, wherein the circuitry is further configured to:
control a moving range of the movable apparatus based on at least the current location information.

### Claim at position 8 (depends on 1)

8: The movable apparatus according to claim 1, wherein the circuitry is further configured to:
control a moving range of the movable apparatus based on at least the feedback information.

### Claim at position 9 (depends on 8)

9: The movable apparatus according to claim 8, wherein the movable apparatus is controlled based on the moving range of the movable apparatus.

### Claim at position 10 (depends on 1)

10: The movable apparatus according to claim 1, wherein the circuitry is further configured to:
obtain a distance from the movable apparatus to the target object by a distance sensor of the movable apparatus.

### Claim at position 11 (depends on 10)

11: The movable apparatus according to claim 10, wherein the circuitry is further configured to:
calculate location information of the target object based on the current location information of the movable apparatus and the distance.

### Claim at position 12 (depends on 1)

12: The movable apparatus according to claim 1, wherein the movable apparatus is controlled based on location information of the target object.

### Claim at position 13

13: A method for operating a movable apparatus comprising:
controllably driving a movement of the movable apparatus;
capturing with a camera a captured image and producing image data;
generating GPS information as current location information of the movable apparatus;
transmitting to an instructing device the current location information of the movable apparatus and the image data, the instructing device displaying a map indicating a current location along a planned travel route of the movable apparatus based on the current location information and a visual indicator that indicates a candidate target object in the display based on the image data;
receiving from the instructing device feedback information that indicates the candidate target object is correctly identified as the target object in the captured image, the feedback information relating to detected user gesture information that is received to control an operation of the movable apparatus, the visual indicator that indicates the target object based on the feedback information; and
controlling the movement of the movable apparatus and an operation of the camera based on the feedback information.

### Claim at position 14 (depends on 13)

14: The method according to claim 13, wherein the movable apparatus is controlled to move towards the target object.

### Claim at position 15 (depends on 13)

15: The method according to claim 13, further comprising:
continuously obtaining image data of the captured image, the captured image including the target object recognized based on the feedback information.

### Claim at position 16 (depends on 15)

16: The method according to claim 15, further comprising:
continuously transmitting the image data of the image including the target object.

### Claim at position 17 (depends on 13)

17: The method according to claim 13, further comprising:
controlling driving circuitry of the movable apparatus to move the movable apparatus.

### Claim at position 18 (depends on 13)

18: The method according to claim 13, further comprising:
controlling a moving range of the movable apparatus based on at least the current location information.

### Claim at position 19 (depends on 13)

19: The method according to claim 13, further comprising:
controlling a moving range of the movable apparatus based on at least the feedback information.

### Claim at position 20

20: A non-transitory computer readable medium having a computer program recorded thereon, the computer program configured to perform a method when executed on a computer, the method comprising:
controllably driving a movement of a movable apparatus;
capturing with a camera a captured image and producing image data;
generating GPS information as current location information of the movable apparatus;
transmitting to an instructing device the current location information of the movable apparatus and the image data, the instructing device displaying a map indicating a current location along a planned travel route of the movable apparatus based on the current location information and a visual indicator that indicates a candidate target object in the based on the image data;
receiving from the instructing device feedback information that indicates the candidate target object is correctly identified as the target object in the captured image, the feedback information relating to detected user gesture information that is received to control an operation of the movable apparatus, the visual indicator that indicates the target object based on the feedback information; and
controlling the movement of the movable apparatus and an operation of the camera based on the feedback information.
