System and method for ontology guided indoor scene understanding for cognitive robotic tasks

A system comprising:

• an input/output interface to obtain at least one navigation command for a robot from a user as an input;
• one or more hardware processors;
• a memory in communication with the one or more hardware processors, wherein the one or more hardware processors are configured to execute programmed instructions stored in the memory, to:
• capture one or more images of a scene by the robot, wherein each of the one or more images is a combination of Red Green Blue (RGB) images and their corresponding depth image;
• identify one or more objects within the scene using a combination of image processing techniques, wherein the image processing techniques comprising an object detection technique, a captioning technique, and an ontology validation technique;
• query an ontology to determine one or more properties related to each of the one or more identified objects;
• select at least one image processing technique from the combination of image processing techniques based on the determined one or more properties of each of the one or more objects and representation of the ontology to extract one or more attributes and relations from the scene;
• generate at least one scene graph using the extracted one or more attributes and relations to aid the robot in executing at least one navigation command;
• and
• update a knowledge graph representation of a knowledge base based on the generated scene graph to facilitate an effective interaction between the user and the robot, wherein the knowledge graph representation represents a grounded representation.

A non-transitory computer readable medium storing one or more instructions which when executed by one or more processors on a system, cause the one or more processors to perform method comprising:

• obtaining, via an input/output interface, at least one navigation command for a robot from a user as an input;
• capturing, via one or more hardware processors, one or more images of a scene by the robot, wherein each of the one or more images is a combination of Red Green Blue (RGB) image and its corresponding depth image;
• identifying, via the one or more hardware processors, one or more objects within the scene using a combination of image processing techniques, comprising an object detection, a captioning, and an ontology validation;
• querying, via the one or more hardware processors, an ontology to determine one or more properties related to each of the one or more identified objects;
• selecting, via the one or more hardware processors, at least one image processing technique from the combination of image processing techniques based on the determined one or more properties of each of the one or more objects and representation of the ontology to extract one or more attributes and relations from the scene;
• generating, via the one or more hardware processors, at least one scene graph using the extracted one or more attributes and relations to aid the robot in executing at least one navigation command;
• and
• updating, via the one or more hardware processors, a knowledge graph representation of a knowledge base based on the generated scene graph to facilitate an effective interaction between the user and the robot, wherein the knowledge graph representation represents a grounded representation.