# US 2019/0171881 A1 · Selectively downloading targeted object recognition modules > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171881 A1 - Document ID: US20190171881A1 - Status: Published application - Kind: A1 - Application number: 16/269,183 - Filed: 2019-02-06 - Published: 2019-06-06 - Earliest claimed: 2016-08-06 - Assignees: X Development LLC - Applicants: X Development LLC - Inventors: Nareshkumar Rajkumar; Stefan Hinterstoisser; Max Bajracharya - Web page: https://patentgrep.com/patent/US20190171881A1 ## CPC - G06V 10/95: {structured as a network, e.g. client-server architectures} - B25J 9/1697: {Vision controlled systems} - G06F 18/2431: Multiple classes - G06T 1/0014: {Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023)} - G06V 10/764: using classification, e.g. of video objects - G06V 10/96: Management of image or video recognition tasks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - H04W 4/025: {using location based information parameters} - H04W 4/60: Subscription-based services using application servers or record carriers, e.g. SIM application toolkits - H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication - Y10S 901/47: Optical - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) - Y10S: TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS ## Abstract Methods, apparatus, systems, and computer-readable media are provided for downloading targeted object recognition modules that are selected from a library of candidate targeted object recognition modules based on various signals. In some implementations, an object recognition client may be operated to facilitate object recognition for a robot. It may download targeted object recognition module(s). Each targeted object recognition module may facilitate inference of an object type or pose of an observed object. The targeted object module(s) may be selected from a library of targeted object recognition modules based on various signals, such as a task to be performed by the robot. The object recognition client may obtain vision data capturing at least a portion of an environment in which the robot operates. The object recognition client may determine, based on the vision data and the downloaded object recognition module(s), information about an observed object in the environment. ## Claims ### Claim 1 A method implemented using one or more processors, comprising: operating, by one or more processors integral with a robot, an object recognition client to facilitate object recognition for the robot; determining an assigned task to be performed by the robot; determining one or more expected objects or object types associated with the assigned task; based on the assigned task and subsequent to the determining, downloading, to the robot by the object recognition client, from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object recognition modules are selected from a library of targeted object recognition modules that is remote from the robot based at least in part on one or more of the expected objects or object types associated with the assigned task to be performed by the robot; obtaining, by the object recognition client, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area; determining, at the robot by the object recognition client, based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and operating the robot to perform the assigned task. ### Claim 2 (depends on 1) The method of claim 1, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of an environment in which the assigned task requires the robot to operate, wherein the one or expected objects or object types are associated with the category of the environment. ### Claim 3 (depends on 1) The method of claim 1, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of a specific object with which the assigned task requires the robot to interact, wherein the one or expected objects or object types are associated with the category of the specific object. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more expected objects or object types associated with the assigned task are determined based on objects encountered by one or more robots historically when performing tasks sharing one or more attributes with the assigned task. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more targeted object recognition modules are selected from the library further based at least in part on available resources of the robot. ### Claim 6 (depends on 5) The method of claim 5, wherein the available resources of the robot include one or more attributes of a wireless signal available to the robot. ### Claim 7 (depends on 1) The method of claim 1, further comprising selecting, by the object recognition client, the one or more targeted object recognition modules from the library. ### Claim 8 (depends on 1) The method of claim 1, wherein the one or more targeted object recognition modules are selected from the library by the remote computing system based on one or more of the expected objects or types associated with the assigned task. ### Claim 9 A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform the following operations: operating, by one or more processors integral with a robot, an object recognition client to facilitate object recognition for the robot; determining an assigned task to be performed by the robot; determining one or more expected objects or object types associated with the assigned task; based on the assigned task and subsequent to the determining, downloading, to the robot by the object recognition client, from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object recognition modules are selected from a library of targeted object recognition modules that is remote from the robot based at least in part on one or more of the expected objects or object types associated with the assigned task to be performed by the robot; obtaining, by the object recognition client, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area; determining, at the robot by the object recognition client, based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and operating the robot to perform the assigned task. ### Claim 10 (depends on 9) The system of claim 9, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of an environment in which the assigned task requires the robot to operate, wherein the one or expected objects or object types are associated with the category of the environment. ### Claim 11 (depends on 9) The system of claim 9, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of a specific object with which the assigned task requires the robot to interact, wherein the one or expected objects or object types are associated with the category of the specific object. ### Claim 12 (depends on 9) The system of claim 9, wherein the one or more expected objects or object types associated with the assigned task are determined based on objects encountered by one or more robots historically when performing tasks sharing one or more attributes with the assigned task. ### Claim 13 (depends on 9) The system of claim 9, wherein the one or more targeted object recognition modules are selected from the library further based at least in part on available resources of the robot. ### Claim 14 (depends on 13) The system of claim 13, wherein the available resources of the robot include one or more attributes of a wireless signal available to the robot. ### Claim 15 (depends on 9) The system of claim 9, further comprising instructions for selecting, by the object recognition client, the one or more targeted object recognition modules from the library. ### Claim 16 (depends on 9) The system of claim 9, wherein the one or more targeted object recognition modules are selected from the library by the remote computing system based on one or more of the expected objects or types associated with the assigned task. ### Claim 17 At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: operating, by one or more processors integral with a robot, an object recognition client to facilitate object recognition for the robot; determining an assigned task to be performed by the robot; determining one or more expected objects or object types associated with the assigned task; based on the assigned task and subsequent to the determining, downloading, to the robot by the object recognition client, from the remote computing system, one or more targeted object recognition modules, wherein each targeted object recognition module is usable by the object recognition client to calculate, at the robot, an object type or pose of an observed object, and wherein the one or more targeted object recognition modules are selected from a library of targeted object recognition modules that is remote from the robot based at least in part on one or more of the expected objects or object types associated with the assigned task to be performed by the robot; obtaining, by the object recognition client, from one or more vision sensors integral with the robot, vision data capturing, from a perspective of the robot, at least a portion of the second area; determining, at the robot by the object recognition client, based on the vision data and the one or more downloaded object recognition modules, information about an object in the second area that is detected in the vision data captured by the one or more vision sensors integral with the robot; and operating the robot to perform the assigned task. ### Claim 18 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of an environment in which the assigned task requires the robot to operate, wherein the one or expected objects or object types are associated with the category of the environment. ### Claim 19 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein determining one or more expected objects or object types associated with the assigned task includes determining a category of a specific object with which the assigned task requires the robot to interact, wherein the one or expected objects or object types are associated with the category of the specific object. ### Claim 20 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein the one or more expected objects or object types associated with the assigned task are determined based on objects encountered by one or more robots historically when performing tasks sharing one or more attributes with the assigned task.