US 11,907,841 B1Granted

Machine learning based consumer product identification system and method therefor

OwnerIan Truitnerinventor

Life of this patent

Estimating the term…

Loading patent…

Abstract

A machine learning based system and method for automated recognition of consumer products in images and video using a camera system, a neural network using a ranked tagging system, a two-stage recognition application, and a training module. Training image sets of items captured by the camera system are assigned identification tags through template matches to training sets within the neural network. Tags are assigned from various levels of specificity to identify exact product matches. A user recognition application captures images and generates bounding boxes for detected objects and assigns a general classification within the image using a single, fast, convolutional neural network (CNN) layer. General classification narrows subsets for each generated bounding box and multi-scale template matching is applied to achieve detailed identification of single or multiple items detected within single images or video. The training module adjusts smart camera systems and the neural network based on accuracy feedback.

Claims21

Dependent claims

A consumer product identification system comprising:

a camera system configured to create an image set of a known consumer product, wherein the camera system creates the image set of the known consumer product by:
placing the known consumer product on a rotating platform;
taking one image of the known consumer product at every 30° of horizontal rotation of the rotating platform resulting in twelve initial images;
tilting the known consumer product by 45° onto its side on the rotating platform;
and
taking one image of the known consumer product at every 30° of horizontal rotation of the rotating platform resulting in twelve additional images and a total of twenty-four images in the image set of the known consumer product;
a neural network configured to:
receive the image set of the known consumer product from the camera system;
store the image set as a template image set in a database connected to the neural network;
assign a plurality of identification tags to the template image set of the known consumer product, wherein the plurality of identification tags is based on characteristics of the known consumer product;
and
generate a machine learning model which is trained based on the assigned set of identification tags to output the known consumer product;
a recognition application configured to:
receive an input image of an unknown consumer product;
compare the input image of the unknown consumer product to the template image set of the known consumer product;
determine whether the input image of the unknown consumer product matches the template image set of the known consumer product;
and
identify the unknown consumer product when the image of the unknown consumer product matches the template image set of the known consumer product;
and
a training module configured to:
receive feedback from a system user regarding accuracy of identification of the unknown consumer product;
adjust settings in at least one of the camera system, the neural network, and the recognition application based on the feedback received from the system user.

19 more elements

A method for identifying consumer products comprising the steps of:

providing a camera system, the camera system creating an image set of a known consumer product for storage on a database, the image set comprising a plurality of images having a pixel size of 3840×2160 pixels;
providing a neural network, the neural network:
selecting five core images from the image set of the known consumer product;
compressing each of the five core images to a square 1080×1080-pixel frame;
designating the five core images as a template image set for template matching;
assigning a plurality of identification tags to the template image set, wherein the plurality of identification tags is based on characteristics of the known consumer product;
and
generating a machine learning model which is trained based on the assigned plurality of identification tags to output the known consumer product;
and
providing a recognition application, the recognition application:
receiving an input image of at least one unknown consumer product;
assigning a general classification to the at least one unknown consumer product within the input image;
selecting one template image set from a plurality of template images sets stored on the database, wherein the selected template image set belongs to the same general classification as the at least one unknown consumer product within the input image;
applying a template matching algorithm to compare the input image of the at least one unknown consumer product to the selected template image set;
determining whether a match between the input image of the at least one unknown consumer product and the selected template image set is greater than 99%, between 90% and 99%, or less than 90% accurate;
and
sending information relating to the at least one unknown consumer product to an electronic device of a consumer when the match between the input image of the at least one unknown consumer product and the template image set of the known consumer product is greater than 90%, between 90% and 99%.

14 more elements

A non-transitory computer-readable medium having stored thereon computer executable instructions that when executed by a computer, causes the computer to execute operations, the operations comprising:

receiving an image set associated with each of a plurality of known consumer products, the image set being created by a camera system and received by a neural network from the camera system;
storing the image set as a template image set in a database connected to the neural network:
generating identification tags associated with characteristics of each of the known consumer products, the identification tags being generated by the neural network;
assigning a set of identification tags to the template image set of each of the known consumer products, the set of identification tags being assigned by the neural network;
generating by the neural network a machine learning model which is trained based on the assigned set of identification tags to output one consumer product from the plurality of known consumer products;
receiving an input image of an unknown consumer product, the input image being received by a recognition application;
comparing by the recognition application the input image of the unknown consumer product to the template image sets of the known consumer products;
determining by the recognition application whether the input image of the unknown consumer product matches the template image set of one of the known consumer products, wherein determining whether the input image of the unknown consumer product matches the template image set of the known consumer product comprises one of the steps of:
determining that an exact match exists between the input image of the unknown consumer product and the template image set of one of the plurality of known consumer products;
determining that a close match exists between the input image of the unknown consumer product and the template image set of one of the plurality of known consumer products;
and
determining that no match exists between the input image of the unknown consumer product and the template image set of one of the plurality of known consumer products;
identifying the unknown consumer product when the image of the unknown consumer product matches the template image set of one of the plurality of known consumer products;
the unknown consumer product being identified by the recognition application;
receiving feedback from a system user regarding accuracy of identification of the unknown consumer product, the feedback being received by a training module;
and
adjusting settings in at least one of the camera system, the neural network, and the recognition application based on the feedback received by the training module from the system user.

14 more elements

A consumer product identification system comprising:

a camera system configured to create an image set of a known consumer product;
a neural network configured to:
receive the image set of the known consumer product from the camera system;
store the image set as a template image set in a database connected to the neural network;
assign a plurality of identification tags to the template image set of the known consumer product, wherein the plurality of identification tags is based on characteristics of the known consumer product;
and
generate a machine learning model which is trained based on the assigned set of identification tags to output the known consumer product;
a recognition application configured to:
receive an input image of an unknown consumer product;
compare the input image of the unknown consumer product to the template image set of the known consumer product;
determine whether the input image of the unknown consumer product matches the template image set of the known consumer product by determining one of:
that an exact match exists between the input image of the unknown consumer product and the images associated with one of the plurality of known consumer products;
that a close match exists between the input image of the unknown consumer product and the images associated with one of the plurality of known consumer products;
and
that no match exists between the input image of the unknown consumer product and the images associated with the one of the plurality of known consumer products;
and
identify the unknown consumer product when the image of the unknown consumer product exactly matches or closely matches the template image set of the known consumer product;
and
a training module configured to:
receive feedback from a system user regarding accuracy of identification of the unknown consumer product;
adjust settings in at least one of the camera system, the neural network, and the recognition application based on the feedback received from the system user.

18 more elements

Dependencies are inferred from the claim text.