System and method for cognitive surveillance robot for securing indoor spaces

A system, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• receive a stream of input images specific to a plurality of scenes related to an indoor space from a robot mounted camera;
• pre-process the received stream of input images by:
• (i) resizing the received stream of input images to a pre-specific optimal dimension;
• (ii) modifying the resized stream of input images based on a plurality of image spatial variations and a plurality of pixel level variations including a hue, a texture, and a gradient;
• and
• (iii) extracting one or more regions of interests from the modified stream of input images using a region of interest extractor network;
• pass the pre-processed stream of input images to a first trained model to obtain a plurality of bounding boxes and one or more masks pertaining to one or more objects comprised in the pre-processed stream of input images, wherein the first trained model ranks the extracted one or more region of interests in order of confidence towards an object class, and wherein the first trained model is obtained by merging a manually annotated dataset and a labelled dataset, and training using a network comprising a region of interest extractor network and a deep network;
• run one or more object property detector algorithms on the one or more masks pertaining to the one or more objects comprised in the stream of input images to detect a plurality of properties of the one or more objects comprised in the pre-processed stream of input images;
• classify one or more signages comprised in the one or more masks pertaining to the one or more objects based on the properties of the one or more objects;
• perform:
• (i) extracting a plurality of regions of interests around the one or more signages pertaining to the one or more objects;
• (ii) passing an image form of a text signage to an optical character recognition (OCR) to detect a language of the text signage with a high affinity to English language, wherein the image form of the text signage is obtained by finding similarity of words comprised in the text signage with one or more templates in each language to find the closest language;
• (iii) detecting one or more texts embedded in the image form of the text signage, wherein the detected one or more texts is assigned a pre-specified cluster tag among a group of concepts based on a similarity matching technique in a word-vector space;
• (iv) passing the one or more signages comprised in the one or more masks pertaining to the one or more objects to a second trained model, to extract a plurality of semantic attributes of a symbol comprised in the one or more signages, and wherein the second trained model is trained on a plurality of different symbols comprised in the indoor space;
• and
• (v) matching the extracted plurality of semantic attributes of the symbol with a knowledgebase to detect an image-based symbol, wherein the knowledgebase comprises textual description and a meaning specific to the matched plurality of semantic attributes of the symbol;
• combine (i) an output of the optical character recognition (OCR) which includes the detected one or more texts embedded in the image form of the text signage and (ii) an output of the second trained model which includes matched plurality of semantic attributes of the symbol;
• convert the combined (i) output of the optical character recognition (OCR) and (ii) the output of the second trained model into one or more facts as triples of subject-predicate-object using a Resource Description Format (RDF) technique;
• infer one or more new facts from the converted one or more facts using a steam reasoning technique, based on one or more reasoning on one or more prior sensed facts and one or more current streaming sensed facts, having an expiry time, and the knowledgebase;
• and
• match one or more patterns comprised in the form of an adaptable business logic with the inferred one or more new facts using registered one or more SPARQL Protocol and Resource Description Format (RDF) Query Language (SPARQL) queries and triggering one or more alerts when the robot moves around to new view angles to inspect correctness of the one or more inferred new facts, and then transfer the alert, avoiding duplicates, avoiding duplicates.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

• receiving a stream of input images specific to a plurality of scenes related to an indoor space from a robot mounted camera;
• pre-processing the received stream of input images:
• (i) resizing the received stream of input images to a pre-specific optimal dimension;
• (ii) modifying the resized stream of input images based on a plurality of image spatial variations and a plurality of pixel level variations including a hue, a texture, and a gradient;
• and
• (iii) extracting one or more regions of interests from the modified stream of input images using a region of interest extractor network;
• passing the pre-processed stream of input images to a first trained model to obtain a plurality of bounding boxes and one or more masks pertaining to one or more objects comprised in the pre-processed stream of input images, wherein the first trained model ranks the extracted one or more regions of interests in order of confidence towards an object class, and wherein the first trained model is obtained by merging a manually annotated dataset and a labelled dataset, and training using a network comprising a region of interest extractor network and a deep network;
• running one or more object property detector algorithms on the one or more masks pertaining to the one or more objects comprised in the pre-processed stream of input images to detect a plurality of properties of the one or more objects comprised in the pre-processed stream of input images;
• classifying one or more signages comprised in the one or more masks pertaining to the one or more objects based on the properties of the one or more objects;
• performing:
• (i) extracting a plurality of regions of interests around the one or more signages pertaining to the one or more objects;
• (ii) passing an image form of a text signage to an optical character recognition (OCR) to detect a language of the text signage with a high affinity to English language, wherein the image form of the text signage is obtained by finding similarity of words comprised in the text signage with one or more templates in each language to find the closest language;
• (iii) detecting one or more texts embedded in the image form of the text signage, wherein the detected one or more texts is assigned a pre-specified cluster tag among a group of concepts based on a similarity matching technique in a word-vector space;
• (iv) passing the one or more signages comprised in the one or more masks pertaining to the one or more objects to a second trained model, to extract a plurality of semantic attributes of a symbol comprised in the one or more signages, and wherein the second trained model is trained on a plurality of symbols comprised in the indoor space;
• and
• (v) matching the extracted plurality of semantic attributes of the symbol with a knowledgebase to detect an image-based symbol, wherein the knowledgebase comprises textual description and a meaning specific to the matched plurality of semantic attributes of the symbol;
• combining (i) an output of the optical character recognition (OCR) which includes the detected one or more texts embedded in the image form of the text signage and (ii) an output of the second trained model which includes matched plurality of semantic attributes of the symbol;
• converting the combined (i) output of the optical character recognition (OCR) and (ii) the output of the second trained model into one or more facts as triples of subject-predicate-object using a Resource Description Format (RDF) technique;
• inferring one or more new facts from the converted one or more facts using a steam reasoning technique, based on one or more reasoning on one or more prior sensed facts and one or more current streaming sensed facts, having an expiry time, and the knowledgebase;
• and
• matching one or more patterns comprised in the form of an adaptable business logic with the one or more inferred new facts using registered one or more SPARQL Protocol and Resource Description Format (RDF) Query Language (SPARQL) queries and triggering one or more alerts, when the robot moves around to new view angles to inspect correctness of the one or more inferred new facts, and then transfer the alert avoiding duplicates.