# US 12,103,190 B1 · Multimodal object identification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,103,190 B1 - Document ID: US12103190B1 - Status: Granted - Kind: B1 - Application number: 17/222,263 - Filed: 2021-04-05 - Granted: 2024-10-01 - Earliest claimed: 2017-07-06 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Michael Joseph Quinlan; Gabriel A. Cohen - Web page: https://patentgrep.com/patent/US12103190B1 ## CPC - B25J 9/1697: {Vision controlled systems} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means} - B25J 11/008: {Manipulators for service tasks} - B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)} - G05B 19/048: Monitoring; Safety - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - B25J 19/026: {Acoustical sensing devices} - G05B 2219/23021: Gesture programming, camera sees hand, displays it on screen, grasp buttons - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Methods, systems, and apparatus for receiving a command for controlling a robot, the command referencing an object, receiving sensor data for a portion of an environment of the robot, identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data, searching map data for the object, determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region, and in response to determining that the object referenced in the command is present in the spatial region, controlling the robot to perform an action with respect to the object referenced in the command. ## Claims ### Claim 1 A computer-implemented method comprising: storing object data that indicates objects in an environment of a robot; detecting, based on sensor data that is captured by a robot and that describes a first region, a gesture of a human that indicates a second region outside the first region described by the sensor data; accessing a portion of the stored object data indicating locations of one or more objects in the second region; identifying, based at least on the accessed portion of the stored object data, an object that that is located in the second region indicated by the gesture; and causing the robot to perform an action associated with the identified object that is located in the second region. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data comprises image data generated using a camera of the robot, and wherein the gesture points to the second region, the second region being a spatial region that is not shown in the image data. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the stored data comprises (i) map data that indicates coordinates of the one or more objects and (ii) data indicating an object type for each of the one or more objects. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the stored data comprises records of objects that the robot previously observed in the second region. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein identifying the object is based on a transcription of one or more words spoken by the human. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein the identified gesture is one of an arm wave, a hand gesture, or a glance. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, comprising: determining a location of the robot in a space; and determining the second region based at least on the gesture of the human and the location of the robot within the space. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, comprising: determining an orientation of the robot when the sensor data is captured; and determining the second region based at least on the gesture of the human and the orientation of the robot. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the second region is a subset of an environment of the robot. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the action associated with the identified object is an action for the robot to retrieve the identified object. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein identifying the object comprises selecting, from among multiple objects of a type indicated by a voice input, a particular object based on a determination that the particular object is located in the second region. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the second region comprises: detecting one or more predetermined shapes from the sensor data, each of the one or more predetermined shapes corresponding to a gesture of a human; determining one or more locations of the detected one or more predetermined shapes within the sensor data; and determining the second region based at least on the one or more predetermined shapes and the one or more locations of the detected one or more predetermined shapes within the sensor data. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, comprising determining a location of the identified object within the second region. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein the location of the object referenced in the voice input within the second region is represented by a set of coordinates. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data includes at least one of image data, infrared image data, light detection and ranging (LIDAR) data, thermal image data, night vision image data, or motion data. ### Claim 16 (depends on 1) The computer-implemented method of claim 1, wherein the robot is configured to obtain image data for a field of view of a camera of the robot, and wherein the second region indicated by the gesture is at least partially outside the field of view of the camera of the robot when the voice input is received. ### Claim 17 A system comprising: one or more sensors; one or more processors; and one or more machine-readable media storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising: storing object data that indicates objects in an environment of a robot; detecting, based on sensor data that is captured by a robot and that describes a first region, a gesture of a human that indicates a second region outside the first region described by the sensor data; accessing a portion of the stored object data indicating locations of one or more objects in the second region; identifying, based at least on the accessed portion of the stored object data, an object that that is located in the second region indicated by the gesture; and causing the robot to perform an action associated with the identified object that is located in the second region. ### Claim 18 (depends on 17) The system of claim 17, wherein the sensor data comprises image data generated using a camera of the robot, and wherein the gesture refers to the second region, the second region being a region that is not shown in the image data. ### Claim 19 (depends on 17) The system of claim 17, wherein the stored data comprises (i) map data that indicates coordinates of the one or more objects and (ii) data indicating an object type for each of the one or more objects. ### Claim 20 One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: storing object data that indicates objects in an environment of a robot; detecting, based on sensor data that is captured by a robot and that describes a first region, a gesture of a human that indicates a second region outside the first region described by the sensor data; accessing a portion of the stored object data indicating locations of one or more objects in the second region; identifying, based at least on the accessed portion of the stored object data, an object that that is located in the second region indicated by the gesture; and causing the robot to perform an action associated with the identified object that is located in the second region.