# US 10,500,720 B2 · Display control device, display control method, computer program product, and communication system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,500,720 B2 - Document ID: US10500720B2 - Status: Granted - Kind: B2 - Application number: 15/722,102 - Filed: 2017-10-02 - Published: 2018-01-25 - Granted: 2019-12-10 - Earliest claimed: 2011-08-02 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Kenta Kawamoto; Yoshiaki Iwai; Satoru Shimizu - Web page: https://patentgrep.com/patent/US10500720B2 ## CPC - B25J 9/16: Program controls (total factory control, i.e. centrally controlling a plurality of machines, G05B19/418) - B25J 5/007: {mounted on wheels} - B25J 9/0003: {Home robots, i.e. small robots for domestic use} - B25J 9/1661: {characterised by task planning, object-oriented languages} - B25J 9/1664: {characterised by motion, path, trajectory planning} - B25J 9/1697: {Vision controlled systems} - B25J 13/06: Control stands, e.g. consoles, switchboards - B25J 13/08: by means of sensing devices, e.g. viewing or touching devices - B25J 13/084: {Tactile sensors (in general G01L5/16, G01L5/22)} - G05B 2219/40099: Graphical user interface for robotics, visual robot user interface - G05B 2219/40123: Indicate, select features on display, remote manipulator will execute - G05B 2219/40409: Robot brings object near operator, operator places object in correct position - G05B 2219/40411: Robot assists human in non-industrial environment like home or office - Y10S 901/01: Mobile robot - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract A control system, method and computer program product cooperate to assist control for an autonomous robot. An interface receives recognition information from an autonomous robot, said recognition information including candidate target objects to interact with the autonomous robot. A display control unit causes a display image to be displayed on a display of candidate target objects, wherein at least two of the candidate target objects are displayed with an associated indication of a target object score. ## Claims ### Claim 1 A control device for a movable apparatus that includes at least one actuator driving motor to move the movable apparatus, the control device comprising: circuitry configured to control a display to display (i) an image on a first region of the display, the image corresponding to a captured image captured by a camera attached on the movable apparatus, and display on a second region of the display (ii) an overhead map including a marker that identifies a present position of the movable apparatus and visual information that extends between the first region and the second region and indicating, in relation to a fixed position, a correspondence between the captured image displayed in the first region and a position of the captured image as shown in the overhead map, wherein the circuitry is configured to control the display to simultaneously display the captured image and the overhead map. ### Claim 2 (depends on 1) The control device according to claim 1, wherein the circuitry is configured to control the display to display the captured image in the first region of the display and to display the overhead map in the second region of the display, the first region being distinct and separate from the second region. ### Claim 3 (depends on 1) The control device according to claim 1, wherein the circuitry is configured to control to the display to superimpose the marker on the overhead map to identify the position of the movable apparatus. ### Claim 4 (depends on 1) The control device according to claim 1, wherein the circuitry is configured to control the display to display path information on the overhead map, the path information indicating a movement of the movable apparatus. ### Claim 5 (depends on 1) The control device according to claim 1, wherein the captured image includes a target object, and position information of the target object is displayed on the overhead map. ### Claim 6 (depends on 1) The control device according to claim 1, wherein the circuitry is configured to control the display to superimpose a second marker to identify a position of the control device. ### Claim 7 (depends on 4) The control device according to claim 4, wherein the path information indicates that the movement of the movable apparatus is towards a target object that is displayed on the overhead map. ### Claim 8 (depends on 5) The control device according to claim 5, wherein the circuitry is further configured to obtain a recognition information of the target object, and control the display to display an indication of the recognition information associated with the target object. ### Claim 9 (depends on 8) The control device according to claim 8, wherein the indication of the recognition information is a second marker that is superimposed on the captured image and that is disposed adjacently to the target object. ### Claim 10 (depends on 1) The control device according to claim 1, wherein the circuitry is further configured to generate a control signal to control the movable apparatus, wherein the generated control signal restricts a movement of the movable apparatus to be within a predetermined area. ### Claim 11 (depends on 8) The control device according to claim 8, wherein the circuitry is further configured to perform a recognition process on the target object to generate the recognition information of the target object. ### Claim 12 (depends on 10) The control device according to claim 10, wherein the predetermined area is determined based on position information of the movable apparatus and position information of the control device. ### Claim 13 (depends on 10) The control device according to claim 10, wherein the predetermined area is determined based on an input from an interface of the control device. ### Claim 14 (depends on 13) The control device according to claim 13, wherein the interface is a touch panel integrally arranged with the display, and the predetermined area is determined based on a designation of an area on the captured image or the overhead map displayed on the display. ### Claim 15 (depends on 1) The control device according to claim 1, wherein the captured image is compressed in size prior to being displayed on the display. ### Claim 16 (depends on 1) The control device according to claim 1, wherein the captured image is compressed in size prior to being transmitted from the camera. ### Claim 17 (depends on 16) The control device according to claim 16, wherein the captured image is compressed by performing an image compression process on only a part of the captured image. ### Claim 18 (depends on 1) The control device according to claim 1, further comprising: an interface, the interface being implemented by a touch panel that is integrally arranged with the display. ### Claim 19 A method for controlling display of information from a movable apparatus that includes at least one actuator driving motor to move the movable apparatus, the method comprising: controlling a display to display (i) an image on a first region of the display, the image corresponding to a captured image captured by a camera attached on the movable apparatus; controlling the display to display on a second region of the display (ii) an overhead map including a marker that identifies a present position of the movable apparatus and visual information that extends between the first region and the second region and indicating, in relation to a fixed position, correspondence between the captured image displayed in the first region and a position of the captured image as shown in the overhead map; and controlling the display to simultaneously display the captured image and the overhead map. ### Claim 20 (depends on 19) The control method according to claim 19, further comprising: controlling the display to display the captured image in a first region of the display and to display the overhead map in the second region of the display, the first region being distinct and separate from the second region. ### Claim 21 (depends on 19) The control method according to claim 19, further comprising: controlling the display to superimpose the marker on the overhead map to identify the position of the movable apparatus. ### Claim 22 (depends on 19) The control method according to claim 19, further comprising: controlling the display to display path information on the overhead map, the path information indicating a movement of the movable apparatus. ### Claim 23 (depends on 19) The control method according to claim 19, wherein the captured image includes a target object, and position information of the target object is displayed on the overhead map. ### Claim 24 (depends on 19) The control method according to claim 19, further comprising: controlling the display to superimpose a second marker to identify a position of the control device. ### Claim 25 (depends on 22) The control method according to claim 22, wherein the path information indicates that the movement of the movable apparatus is towards a target object that is displayed on the overhead map. ### Claim 26 (depends on 23) The control method according to claim 23, further comprising: obtaining a recognition information of the target object; and controlling the display to display an indication of the recognition information associated with the target object. ### Claim 27 (depends on 26) The control method according to claim 26, wherein the indication of the recognition information is a second marker that is superimposed on the captured image and that is disposed adjacently to the target object. ### Claim 28 (depends on 19) The control method according to claim 19, further comprising: generating a control signal to control the movable apparatus, wherein the generated control signal restricts a movement of the movable apparatus to be within a predetermined area. ### Claim 29 (depends on 26) The control method according to claim 26, further comprising: performing a recognition process on the target object to generate the recognition information of the target object. ### Claim 30 (depends on 28) The control method according to claim 28, further comprising: determining the predetermined area based on position information of the movable apparatus and position information of the control device. ### Claim 31 (depends on 28) The control method according to claim 28, further comprising: determining the predetermined area based on an input from an interface of the control device. ### Claim 32 (depends on 31) The control method according to claim 31, further comprising: determining the predetermined area based on a designation of an area on the captured image or the overhead map displayed on the display, wherein the interface is a touch panel integrally arranged with the display. ### Claim 33 (depends on 19) The control method according to claim 19, further comprising: compressing the captured image in size prior to being displayed on the display. ### Claim 34 (depends on 19) The control method according to claim 19, further comprising: compressing the captured image in size prior to being transmitted from the camera. ### Claim 35 (depends on 34) The control method according to claim 34, further comprising: compressing the captured image by performing an image compression process on only a part of the captured image. ### Claim 36 (depends on 19) The control method according to claim 19, further comprising: implementing an interface by a touch panel that is integrally arranged with the dsplay.