Back to results
US 12,749,041 B2Granted
Systems and methods for identifying exceptions in feature detection analytics
AssigneeBrain Corporation
InventorsAndrew Hekman, Mingwei Xu, Kiel Martin Groeniger
Life of this patent
Estimating the term…
Abstract
Systems and methods for identifying exceptions in feature detection analytics include systems and methods configured to identify exceptions in product displays. In one exemplary embodiment, price tag mismatches are reported to a customer if certain criteria are met based on analytics from optical character recognition, image object detection, and reference catalogs/planograms. These price tag mismatches, for example, provide actionable insights for humans working alongside robots which can be quickly identified and resolved.
Claims23
A system configured to identify exceptions in product displays, comprising:
- a robot comprising at least one sensor configured to take images of objects as the robot travels in an environment, each image acquired by the robot is localized by a controller thereon;
- at least one user device;
- and
- a server in communication with the robot and at least one user device, the server comprises at least one processor configured to execute computer readable instructions to:
- receive the images of objects and associated localization data for the images from the robot, the images depict at least one feature to be identified, the images corresponded to a location of the robot measured by the robot during acquisition of the images;
- generate a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
- generate a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
- receive a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
- identify at least one exception, the at least one exception being any one or more of:
- (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets and products in the catalog;
- (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
- generate a report comprising a list of identified features and the at least one exception;
- and
- communicate the report to a user device.
11 more elements
A method for identifying exceptions in product displays, comprising:
- a server comprising at least one processor configured to execute computer readable instructions to perform steps of:
- receiving an image of objects and associated localization data for the image from a robot comprising at least one sensor configured to take images of objects as the robot travels in an environment and a controller configured to localize the image as it is taken, wherein the image depicts at least one feature to be identified, the images corresponded to a location of the robot measured by the robot during acquisition of the images;
- generating a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
- generating a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
- receiving a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
- identifying at least one exception being any one or more of:
- (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets with products in the catalog;
- (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
- generating a report comprising a list of identified features and the at least one exception;
- and
- communicating the report to a user device.
8 more elements
A non-transitory computer readable storage medium having a plurality of computer readable instructions stored thereon which, when executed by at least one processor in communication with a robot, configure the at least one processor to identify exceptions in product displays by executing instructions to,
- receive at least one image and associated localization data from the robot, wherein the robot comprises at least one sensor configured to take images of objects as the robot travels in an environment, wherein each image acquired by the robot is localized by a controller thereon and each image depicts at least one feature to be identified and corresponds to a location of the robot measured by the robot during acquisition of the image;
- generate a first data set comprising image object detection predictions for each of the at least one features to be identified within the image based on visual characteristics of the at least one features using a first detection model, the image object detection predictions include at least a product identification and location for each of the at least one features;
- generate a second data set comprising optical character recognition predictions for each of the at least one features to be identified within the image based on text depicted on price labels within the image using a second detection model, the predictions include at least a price value and a product identification for each of the at least one features;
- receive a third data set corresponding to a catalog, the catalog indicates an expected arrangement, location, and price values of features within the environment;
- identify at least one exception being any one or more of:
- (i) a discrepancy between the product identifications in the first and second data sets, (ii) a discrepancy between the product identifications in the first and second data sets with products in the catalog;
- (iii) a discrepancy between the price values in the second data set with the price values in the catalog;
- generate a report comprising a list of identified features and the at least one exception;
- and
- communicate the report to a user device.
7 more elements
Dependencies are inferred from the claim text.