Method and system for multi-object tracking and navigation without pre-sequencing

A system for multi-object tracking and navigation, further comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• continuously receive a plurality of images of an environment via a sensor controlled of a robot while in motion, wherein each of the plurality of images comprises a plurality of objects present in a field of view of the robot, and receiving by the robot, a user task further comprising a plurality of target objects from among the plurality of objects to be located in the environment;
• iteratively create a semantic map of the environment at every first timestep of each episode by using an odometry pose readings and RGB-D observations of the plurality of objects, wherein the semantic map indicates the robot, an arrangement of a plurality of obstacles, the plurality of objects and an associated vacant space in the environment;
• identify by the robot executed the plurality of target objects from among the plurality of objects by processing the semantic map;
• extract via an encoder network a plurality of high-level feature embeddings of the plurality of target objects by using the semantic map and dynamically identifying change in position of the plurality of target objects and the robot in the environment;
• send the plurality of high-level feature embeddings and a plurality of transition states associated with change in position of the plurality of target objects to an actor-critic network that repetitively computes a long-term goal for each episode, wherein the long-term goal is a region of the semantic map where the robot navigates in search of a first instance of the plurality of target objects;
• receive by the robot from the actor critic network a deterministic local policy to compute a navigational action to navigate along a shortest path calculated from a current location of the robot to the long-term goal to reach the target object among the plurality of target objects;
• navigate from a current location to a next location of the target object from among the plurality of target objects based on the deterministic local policy, wherein a trained deep reinforcement learning network executed by the robot with at least one of a reward among a plurality of rewards for performing a navigational action on reaching each target object location among the plurality of target objects;
• and
• utilize a deep reinforcement learning network trained to assign the robot with a computed reward function when the navigational action is performed by the robot to reach the first instance of the plurality of target objects, wherein the robot ends the task as one of a successful task when the plurality of target objects are successfully navigated by the robot, and a failure task when the semantic map predicts no space to navigate in the environment.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• continuously receiving a plurality of images of an environment via a sensor controlled by one or more hardware processors of a robot while in motion, wherein each of the plurality of images comprises a plurality of objects present in a field of view of the robot, and receiving by the robot, a user task further comprising a plurality of target objects from among the plurality of objects to be located in the environment;
• iteratively creating a semantic map of the environment at every first timestep of each episode by using an odometry pose readings and RGB-D observations of the plurality of objects, wherein the semantic map indicates the robot, an arrangement of a plurality of obstacles, the plurality of objects and an associated vacant space in the environment;
• identifying by the robot executed the plurality of target objects from among the plurality of objects by processing the semantic map;
• extracting via an encoder network a plurality of high-level feature embeddings of the plurality of target objects by using the semantic map and dynamically identifying change in position of the plurality of target objects and the robot in the environment;
• sending the plurality of high-level feature embeddings and a plurality of transition states associated with change in position of the plurality of target objects to an actor-critic network that repetitively computes a long-term goal for each episode, wherein the long-term goal is a region of the semantic map where the robot navigates in search of a first instance of the plurality of target objects;
• receiving by the robot from the actor critic network a deterministic local policy to compute a navigational action to navigate along a shortest path calculated from a current location of the robot to the long-term goal to reach the target object among the plurality of target objects;
• navigating from a current location to a next location of the target object from among the plurality of target objects based on the deterministic local policy, wherein a trained deep reinforcement learning network rewards the robot with at least one of a reward among a plurality of rewards for performing a navigational action on reaching each target object location among the plurality of target objects;
• and
• utilizing a deep reinforcement learning network trained to assign the robot with a computed reward function when the navigational action is performed by the robot to reach the first instance of the plurality of target objects, wherein the robot ends the task as one of a successful task when the plurality of target objects are successfully navigated by the robot, and a failure task when the semantic map predicts no space to navigate in the environment.