# US 10,427,306 B1 · Multimodal object identification > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,427,306 B1 - Document ID: US10427306B1 - Status: Granted - Kind: B1 - Application number: 15/643,138 - Filed: 2017-07-06 - Granted: 2019-10-01 - Earliest claimed: 2017-07-06 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Michael Joseph Quinlan; Gabriel A. Cohen - Web page: https://patentgrep.com/patent/US10427306B1 ## CPC - B25J 9/1697: {Vision controlled systems} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means} - B25J 11/008: {Manipulators for service tasks} - B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)} - G05B 19/048: Monitoring; Safety - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - B25J 19/026: {Acoustical sensing devices} - G05B 2219/23021: Gesture programming, camera sees hand, displays it on screen, grasp buttons - G05B 2219/35444: Gesture interface, controlled machine observes operator, executes commands - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract Methods, systems, and apparatus for receiving a command for controlling a robot, the command referencing an object, receiving sensor data for a portion of an environment of the robot, identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data, accessing map data indicating locations of objects within a space, searching the map data for the object, wherein the search of the map data is restricted to the spatial region, determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region, and in response to determining that the object referenced in the command is present in the spatial region, controlling the robot to perform an action with respect to the object referenced in the command. ## Claims ### Claim 1 A computer-implemented method comprising: receiving a command for controlling a robot, the command referencing an object; receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot; identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data; in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command; searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture; determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the identified gesture is one of an arm wave, a hand gesture, or a glance. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein receiving the command for controlling the robot, the command referencing the object comprises: receiving audio data captured by a microphone of the robot that corresponds to the command; identifying, based on performing speech recognition on the audio data, one or more candidate objects that each correspond to a respective candidate transcription of at least a portion of the audio data; accessing an inventory of one or more objects within the space; and identifying the object from among the inventory of the one or more objects within the space based at least on comparing each candidate transcription of at least a portion of the audio data to the one or more objects of the inventory. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: determining a location of the robot within the space; and determining the spatial region based at least on the gesture of the human and the location of the robot within the space. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: determining an orientation of the robot within the space when receiving the sensor data; and determining the spatial region based at least on the gesture of the human and the orientation of the robot within the space. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: detecting one or more predetermined shapes from the sensor data, each of the one or more predetermined shapes corresponding to a gesture of a human; determining one or more locations of the detected one or more predetermined shapes within the sensor data; and determining the spatial region based at least on the one or more predetermined shapes and the one or more locations of the detected one or more predetermined shapes within the sensor data. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, comprising: receiving a second command for controlling the robot, the command referencing a second object; receiving second sensor data for a portion of the environment of the robot, the second sensor data being captured by the sensor of the robot; identifying, from the second sensor data, a second gesture of a human that indicates a second spatial region located outside of the portion of the environment described by the second sensor data; searching the map data for the second object referenced in the second command, wherein the search of the map data is restricted, based on the identified second gesture, to the second spatial region indicated by the second gesture; determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture; based at least on determining that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture, searching the map data for the second object referenced in the second command, wherein the search of the map is restricted to a third spatial region; determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is present in the third spatial region; and in response to determining that the second object referenced in the second command is present in the third spatial region, controlling the robot to perform a second action with respect to the second object referenced in the second command. ### Claim 8 (depends on 7) The computer-implemented method of claim 7, wherein the third spatial region is larger than the second spatial region indicated by the second gesture. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, comprising: receiving a second command for controlling the robot, the command referencing a second object; receiving second sensor data for a portion of the environment of the robot, the second sensor data being captured by the sensor of the robot; identifying, from the second sensor data, a second gesture of a human that indicates a second spatial region located outside of the portion of the environment described by the second sensor data; searching the map data for the second object referenced in the second command, wherein the search of the map data is restricted, based on the identified second gesture, to the second spatial region indicated by the second gesture; determining, based at least on searching the map data for the second object referenced in the second command, that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture; and based at least on determining that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture, controlling the robot to indicate that the second object referenced in the second command is absent from the second spatial region indicated by the second gesture. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein controlling the robot to perform the action with respect to the object referenced in the command comprises controlling the robot to perform one of retrieving the object referenced in the command, move the object referenced in the command to a predetermined location, move the object referenced in the command to a location indicated by the command, or navigate to the location of the object referenced in the command. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, comprising: determining a location of the object referenced in the command within the spatial region, wherein the location of the object referenced in the command within the spatial region is represented by a set of coordinates within the space. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data is one of image data, infrared image data, light detection and ranging (LIDAR) data, thermal image data, night vision image data, or motion data. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the sensor data for the portion of the environment of the robot is image data for a field of view of a camera of the robot, the image data being captured by the camera of the robot. ### Claim 14 A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a command for controlling a robot, the command referencing an object; receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot; identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data; in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command; searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture; determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command. ### Claim 15 (depends on 14) The system of claim 14, wherein the identified gesture is one of an arm wave, a hand gesture, or a glance. ### Claim 16 (depends on 14) The system of claim 14, wherein receiving the command for controlling the robot, the command referencing the object comprises: receiving audio data captured by a microphone of the robot that corresponds to the command; identifying, based on performing speech recognition on the audio data, one or more candidate objects that each correspond to a respective candidate transcription of at least a portion of the audio data; accessing an inventory of one or more objects within the space; and identifying the object from among the inventory of the one or more objects within the space based at least on comparing each candidate transcription of at least a portion of the audio data to the one or more objects of the inventory. ### Claim 17 (depends on 14) The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: determining a location of the robot within the space; and determining the spatial region based at least on the gesture of the human and the location of the robot within the space. ### Claim 18 (depends on 14) The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: determining an orientation of the robot within the space when receiving the sensor data; and determining the spatial region based at least on the gesture of the human and the orientation of the robot within the space. ### Claim 19 (depends on 14) The system of claim 14, wherein identifying the gesture of the human that indicates the spatial region located outside of the portion of the environment described by the sensor data comprises: detecting one or more predetermined shapes from the sensor data, each of the one or more predetermined shapes corresponding to a gesture of a human; determining one or more locations of the detected one or more predetermined shapes within the sensor data; and determining the spatial region based at least on the one or more predetermined shapes and the one or more locations of the detected one or more predetermined shapes within the sensor data. ### Claim 20 A non-transitory computer-readable storage device storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving a command for controlling a robot, the command referencing an object; receiving sensor data for a portion of an environment of the robot, the sensor data being captured by a sensor of the robot; identifying, from the sensor data, a gesture of a human that indicates a spatial region located outside of the portion of the environment described by the sensor data; in response to identifying the gesture, accessing map data indicating locations of objects within a space, the map data being generated before receiving the command; searching the map data for the object referenced in the command, wherein the search of the map data is restricted, based on the identified gesture, to the spatial region indicated by the gesture; determining, based at least on searching the map data for the object referenced in the command, that the object referenced in the command is present in the spatial region indicated by the gesture; and in response to determining that the object referenced in the command is present in the spatial region indicated by the gesture, controlling the robot to perform an action with respect to the object referenced in the command.