# US 11,029,694 B2 · Self-aware visual-textual co-grounded navigation agent > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,694 B2 - Document ID: US11029694B2 - Status: Granted - Kind: B2 - Application number: 16/176,955 - Filed: 2018-10-31 - Published: 2020-04-02 - Granted: 2021-06-08 - Earliest claimed: 2018-09-27 - Assignees: salesforce.com, inc. - Applicants: salesforce.com, inc. - Inventors: Chih-Yao Ma; Caiming Xiong - Web page: https://patentgrep.com/patent/US11029694B2 ## CPC - G05D 1/0016: {characterised by the operator's input device (input arrangements for computing systems in general G06F3/00)} - B25J 9/0003: {Home robots, i.e. small robots for domestic use} - B25J 9/1697: {Vision controlled systems} - G05D 1/0221: {involving a learning process} - G05D 1/0246: {using a video camera in combination with image processing means} - G05D 1/223: Command input arrangements on the remote controller, e.g. joysticks or touch screens - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 18/256: {of results relating to different input data, e.g. multimodal recognition} - G06N 3/02: Neural networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/764: using classification, e.g. of video objects - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06V 10/82: using neural networks - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An agent for navigating a mobile automated system is disclosed herein. The navigation agent receives a navigation instruction and visual information for one or more observed images. The navigation agent is provided or equipped with self-awareness, which provides or supports the following abilities: identifying which direction to go or proceed by determining the part of the instruction that corresponds to the observed images (visual grounding), and identifying which part of the instruction has been completed or ongoing and which part is potentially needed for the next action selection (textual grounding). In some embodiments, the navigation agent applies regularization to ensures that the grounded instruction can correctly be used to estimate the progress made towards the navigation goal (progress monitoring). ## Claims ### Claim 1 A computing device comprising: a memory containing machine readable medium storing machine executable code; and one or more processors coupled to the memory and configurable to execute the machine executable code to cause the one or more processors to: receive a navigation instruction for instructing a mobile automated system to navigate an environment in which the mobile automated system is located; receive visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generate an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generate a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile automated system should proceed; and using the instruction grounding and the visual grounding, generate an action for the mobile automated system to perform for navigating the environment. ### Claim 2 (depends on 1) The computing device of claim 1, wherein the machine executable code further causes the one or more processors to monitor progress of navigation of the automated system to ensure that the instruction grounding accurately reflects the navigation progress. ### Claim 3 (depends on 1) The computing device of claim 1, wherein the machine executable code causes the one or more processors to: generate an encoder context based on the instruction grounding and the visual grounding; and generate the action for the mobile automated system using the encoder context. ### Claim 4 (depends on 1) The computing device of claim 1, wherein the machine executable code causes the one or more processors to perform a natural language processing task on the navigation instruction. ### Claim 5 (depends on 1) The computing device of claim 1, wherein the machine executable code causes the one or more processors to identify a navigable direction with the highest correlation to the instruction grounding. ### Claim 6 (depends on 1) The computing device of claim 1, wherein the machine executable code causes the one or more processors to: identify a plurality of directions in which the mobile automated system can navigate; and for each identified navigable direction, generate a respective probability. ### Claim 7 A method for navigating a mobile automated system, the method comprising: receiving, at one or more processors, a navigation instruction for instructing the mobile automated system to navigate an environment in which the mobile automated system is located; receiving, at the one or more processors, visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generating, at the one or more processors, an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generating, at the one or more processors, a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile automated system should proceed; and using the instruction grounding and the visual grounding, generating, at the one or more processors, an action for the mobile automated system to perform for navigating the environment. ### Claim 8 (depends on 7) The method of claim 7, comprising monitoring progress of navigation of the automated system to ensure that the instruction grounding accurately reflects the navigation progress. ### Claim 9 (depends on 7) The method of claim 7, wherein generating an action comprises: generating an encoder context based on the instruction grounding and the visual grounding; and generating the action for the mobile automated system using the encoder context. ### Claim 10 (depends on 7) The method of claim 7, comprising performing a natural language processing task on the navigation instruction. ### Claim 11 (depends on 7) The method of claim 7, wherein generating an action comprises identifying a navigable direction with the highest correlation to the instruction grounding. ### Claim 12 (depends on 7) The method of claim 7, wherein generating an action comprises: identifying a plurality of directions in which the mobile automated system can navigate; and for each identified navigable direction, generating a respective probability. ### Claim 13 A non-transitory machine-readable medium comprising executable code which when executed by one or more processors associated with a computing device are adapted to cause the one or more processors to perform a method comprising: receiving, at the one or more processors, a navigation instruction for instructing the mobile automated system to navigate an environment in which the mobile automated system is located; receiving, at the one or more processors, visual information for the environment, the visual information comprising one or more images observed for the environment as the mobile automated system is navigated therethrough; generating, at the one or more processors, an instruction grounding based at least in part on the navigation instruction, the instruction grounding identifying which part of the navigation instruction has been completed by the mobile automated system and which part of the navigation instruction is outstanding; generating, at the one or more processors, a visual grounding based at least in part on the visual information, the visual grounding identifying a direction in which the mobile automated system should proceed; and using the instruction grounding and the visual grounding, generating, at the one or more processors, an action for the mobile automated system to perform for navigating the environment. ### Claim 14 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the executable code further causes the one or more processors to monitor progress of navigation of the automated system to ensure that the instruction grounding accurately reflects the navigation progress. ### Claim 15 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the executable code causes the one or more processors to: generate an encoder context based on the instruction grounding and the visual grounding; and generate the action for the mobile automated system using the encoder context. ### Claim 16 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the executable code causes the one or more processors to perform a natural language processing task on the navigation instruction. ### Claim 17 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the executable code causes the one or more processors to identify a navigable direction with the highest correlation to the instruction grounding. ### Claim 18 (depends on 13) The non-transitory machine-readable medium of claim 13, wherein the executable code causes the one or more processors to: identify a plurality of directions in which the mobile automated system can navigate; and for each identified navigable direction, generate a respective probability.