Retail shelf image processing and inventory tracking system

A system for identifying a product among a first set of products from a shelf image, the system comprising:

• an imaging device;
• a computing system comprising:
• a processor;
• a memory communicatively connected to the processor which stores program instructions executable by the processor, wherein, when executed the program instructions cause the system to:
• receive the shelf image from the imaging device, wherein the shelf image is an image of a shelf that includes the first set of products;
• receive a planogram image of the shelf, wherein the planogram image includes a second set of products;
• compute a visual similarity matrix between the first set of products and the second set of products;
• generate a transformation matrix that maps the first set of products from the shelf image to the second set of products in the planogram image, wherein values of the transformation matrix are derived from values of the visual similarity matrix;
• iteratively refine the transformation matrix for a predetermined number of iterations;
• transform the planogram image by applying the refined transformation matrix;
• compare the transformed planogram image to the shelf image to determine a mapping between the first set of products from the shelf image and the second set of products from the planogram image;
• determine a shelf location of the product within the shelf image;
• based on a mapping of the transformed planogram image to the shelf image, identify a planogram product within the planogram image that is located in a planogram location that corresponds to the shelf location;
• and
• identify the product as the planogram product.

A method for identifying a missing product within a shelf image, the method comprising:

• receiving the shelf image, wherein the shelf image is an image of a shelf that includes a first set of products;
• receiving a planogram image of the shelf, wherein the planogram image includes a second set of products;
• computing a visual similarity matrix between the first set of products and the second set of products;
• generating a transformation matrix that maps the second set of products from the planogram image to the first set of products in the shelf image, wherein values of the transformation matrix are derived from values of the visual similarity matrix;
• iteratively refining the transformation matrix for a predetermined number of iterations, including, for each iteration:
• projecting the planogram image into image space via the transformation matrix;
• computing a distance matrix between the positions of the second set of products from the planogram image and positions the first set of products from the shelf image;
• computing a total similarity matrix between the first set of products from the shelf image and the second set of products from the planogram image as a weighted sum of the visual similarity matrix and the distance matrix;
• based on the total similarity matrix, computing one or more optimal matches between each of the first set of products within the shelf image and the second set of products within the planogram image;
• and
• using the one or more optimal matches re-estimating the values of the transformation matrix;
• transforming the planogram image by applying the refined transformation matrix;
• comparing the transformed planogram image to the shelf image to determine a mapping between the first set of products from the shelf image and the second set of products from the planogram image;
• determining a shelf location of the missing product within the shelf image;
• based on a mapping of the transformed planogram image to the shelf image, identifying a planogram product within the planogram image that is located in a planogram location that corresponds to the shelf location;
• and
• determining that the missing product is the planogram product.