Retrieval-augmented navigation system and method
AssigneeNaver Corporation
InventorsGianluca MONACI, Rafael SAMPAIO DE REZENDE, Romain DEFFAYET, Hervé DÉJEAN, Stéphane CLINCHANT, Gabriela CSURKA KHEDARI, Christian WOLF, Guillaume BONO
Life of this patent
Estimating the term…
Abstract
Methods and systems for performing a goal-oriented visual navigation task for an autonomous device in a three-dimensional navigation environment. An observation image is received. A set of images are retrieved from an image database, and a context is encoded. A current state of an agent is updated based on the encoded context for determining a next action with a learned policy for navigating the agent and controlling movement of the autonomous device.
Claims30
A computer-implemented method for performing a goal-oriented visual navigation task for an autonomous device in a three-dimensional navigation environment, the method comprising:
- receiving an observation image;
- retrieving a set of images from an image database based on at least a goal;
- comparing the retrieved set of images to at least the received observation image;
- encoding a context based on said comparing;
- updating a current state of an agent based on at least the goal, observation image, previous states and actions, and the encoded context;
- determining a next action based on the current state with a learned policy for navigating the agent;
- and
- outputting the determined next action to a control module for controlling movement of the autonomous device.
A navigation system including a retrieval-augmented agent in an autonomous device, the retrieval-augmented agent receiving as an input a goal in a three-dimensional navigation environment, the navigation system comprising:
- an image database comprising a dataset of images;
- a retrieval module that retrieves a set of images from said image database based at least on a goal;
- an embedding module that compares the retrieved set of images to at least a received observation image and generates embeddings for the retrieved set of images;
- a context encoding module that generates a context based on the generated embeddings;
- a navigation module that updates a current state of the retrieval-augmented agent based at least on the context, the goal, a current observation, and previous states and actions;
- and
- an action determining module that determines a next action based on the current state with a learned policy for navigating the agent in the environment and outputs the selected next action to a control module for controlling movement of the autonomous device.
An apparatus comprising:
- a processor and a memory;
- computer-readable instructions stored in said memory and executable by said processor to perform a method for performing a goal-oriented visual navigation task for an autonomous device in a three-dimensional navigation environment, the method comprising:
- receiving an observation image;
- retrieving a set of images from an image database based on at least a goal;
- comparing the retrieved set of images to at least the received observation image;
- encoding a context based on said comparing;
- updating a current state of an agent based on at least the received goal, observation image, previous states and actions, and the encoded context;
- determining a next action based on the current state with a learned policy for navigating the agent towards the location in the navigation environment indicated by the received goal;
- and
- outputting the selected next action to a control module for controlling movement of an autonomous device.
Dependencies are inferred from the claim text.