# US 11,906,973 B2 · Electronic device and method for controlling robot > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,906,973 B2 - Document ID: US11906973B2 - Status: Granted - Kind: B2 - Application number: 17/149,416 - Filed: 2021-01-14 - Published: 2021-07-22 - Granted: 2024-02-20 - Earliest claimed: 2020-01-21 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Jinseon Yoo; Harkjoon Kim; Sungbin Kuk; Jimin Kim; Eunjoo Mo; Chiyoung Ahn; Heeyoung Cho - Web page: https://patentgrep.com/patent/US11906973B2 ## CPC - G05D 1/0234: {using optical markers or beacons (optical beacons per se G01S1/70)} - B25J 9/1697: {Vision controlled systems} - G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)} - G05D 1/0033: {by having the operator tracking the vehicle either by direct line of sight or via one or more cameras located remotely from the vehicle} - G05D 1/223: Command input arrangements on the remote controller, e.g. joysticks or touch screens - G05D 1/247: using signals provided by artificial sources external to the vehicle, e.g. navigation beacons - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G05B 2219/40099: Graphical user interface for robotics, visual robot user interface - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 2200/24: involving graphical user interfaces [GUIs] - G06T 2207/30204: Marker - G06T 2219/2016: Rotation, translation, scaling - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract An electronic device and method for controlling a robot is provided. The electronic device includes a wireless communication unit, a camera, a touch display, a memory, and a processor configured to be operatively connected to the wireless communication unit, the camera, the touch display, and the memory. The processor is configured to recognize a marker of the robot photographed using the camera; generate an indicator indicating a space around the recognized robot; acquire location information for moving the robot to an area within the indicator by using the touch display; and transmit the acquired location information to the robot to move the robot to a location corresponding to the acquired location information. ## Claims ### Claim 1 An electronic device for controlling a robot, the electronic device comprising: a wireless communication unit; a camera; a touch display; a memory; and a processor operatively connected to the wireless communication unit, the camera, the touch display, and the memory, wherein the processor is configured to: recognize a marker of the robot photographed using the camera, generate a grid-shaped indicator of a space around the recognized robot with respect to coordinates at which the robot is located, wherein the grid-shaped indicator divides the space into a plurality of cells including a first cell where the robot is recognized to be located, generate, based on receiving a touch input on a second cell among the plurality of cells on the touch display, a virtual robot on the grid-shaped indicator, acquire location information for moving the robot from the first cell to the second cell, display a predicted movement path through which the robot is moved to a location corresponding to the acquired location information, and transmit the acquired location information to the robot to move the robot to the location corresponding to the acquired location information. ### Claim 2 (depends on 1) The electronic device of claim 1, wherein the processor is further configured to instruct the moved robot to perform a predetermined task on a specific object. ### Claim 3 (depends on 1) The electronic device of claim 1, wherein the location information comprises coordinate information. ### Claim 4 (depends on 1) The electronic device of claim 1, wherein the robot is a mobile robot and comprises at least one of a cleaner, an air cleaner, a drone, or a guide robot. ### Claim 5 (depends on 1) The electronic device of claim 1, wherein the indicator comprises an augmented reality (AR) matrix. ### Claim 6 (depends on 2) The electronic device of claim 2, wherein the processor is further configured to: receive location information of the robot from the electronic device through a server, and provide map information on the specific object to the electronic device or the robot. ### Claim 7 (depends on 6) The electronic device of claim 6, wherein the processor is further configured to cause the indicator to generate location information having absolute coordinates rather than relative coordinates as the map information on the specific object is received through the server. ### Claim 8 (depends on 1) The electronic device of claim 1, wherein the processor is further configured to generate a size and a direction of the indicator based on size and direction information of the recognized robot. ### Claim 9 (depends on 1) The electronic device of claim 1, wherein, as an angle of view of the camera is moved, the processor is further configured to generate an extension region of the indicator in a region corresponding to the moved angle of view of the camera. ### Claim 10 (depends on 1) The electronic device of claim 1, wherein the processor is further configured to: generate the indicator based on a photographed location of the marker photographed using the camera, or generate the indicator in a different form based on a recognized location of the marker photographed using the camera. ### Claim 11 A method of an electronic device for controlling a robot, the method comprising: recognizing a marker of a robot photographed using a camera; generating a grid-shaped indicator of a space around the recognized robot with respect to coordinates at which the robot is located, wherein the grid-shaped indicator divides the space into a plurality of cells including a first cell where the robot is recognized to be located; generating, based on receiving a touch input on a second cell among the plurality of cells on the touch display, a virtual robot on the grid-shaped indicator; acquiring location information for moving the robot from the first cell to the second cell; displaying a predicted movement path through which the robot is moved to a location corresponding to the acquired location information; and moving the robot to the location corresponding to the acquired location information by transmitting the acquired location information to the robot. ### Claim 12 (depends on 11) The method of claim 11, further comprising instructing the moved robot to perform a predetermined task on a specific object. ### Claim 13 (depends on 11) The method of claim 11, wherein the indicator comprises an augmented reality (AR) matrix, and the location information comprises coordinate information. ### Claim 14 (depends on 12) The method of claim 12, further comprising: receiving the location information of the robot from the electronic device through a server; and providing map information on the specific object to the electronic device or the robot by using the server. ### Claim 15 (depends on 11) The method of claim 11, further comprising generating a size and a direction of the indicator based on size and direction information of the recognized robot. ### Claim 16 (depends on 11) The method of claim 11, further comprising generating an extension region of the indicator, as an angle of view of the camera is moved, in a region corresponding to the moved angle of view of the camera. ### Claim 17 (depends on 11) The method of claim 11, wherein generating the indicator further comprises one of: generating the indicator based on a photographed location of the marker photographed using the camera; or generating the indicator in a different form based on a recognized location of the marker photographed using the camera.