# US 11,443,277 B2 · System and method for identifying object information in image or video data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,277 B2 - Document ID: US11443277B2 - Status: Granted - Kind: B2 - Application number: 16/869,723 - Filed: 2020-05-08 - Published: 2021-09-30 - Granted: 2022-09-13 - Earliest claimed: 2020-03-26 - Assignees: Fractal Analytics Private Limited - Applicants: Fractal Analytics Private Limited - Inventors: Prakash Vanapalli; Vikash Challa; Sachin Chandra Bonagiri; Suraj Amonkar; Sarang Pande; Samreen Khan - Web page: https://patentgrep.com/patent/US11443277B2 ## CPC - G06Q 10/06393: {Score-carding, benchmarking or key performance indicator [KPI] analysis} - G06F 16/51: Indexing; Data structures therefor; Storage structures - G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade} - G06F 18/23: Clustering techniques - G06F 18/2431: Multiple classes - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06Q 10/087: Inventory or stock management, e.g. order filling, procurement or balancing against orders - G06Q 10/0877: {by inventory control or reporting using inventory tracking or counting} - G06V 10/762: using clustering, e.g. of similar faces in social networks - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06N 20/00: Machine learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Systems and methods are provided for identifying a product in an image and outputting stock keeping units of the product. The system comprises three main components: a database server, a data analytics system and a standard dashboard. The database server contains real-time inventory images as well as historical images of each product type. The data analytics system is executed by a computer processor configured to apply object detection and classification and deep learning algorithms to detect product information captured by the image. The data analytics system is also configured to determine hierarchical classification categories for the product. The standard dashboard is configured to output a report regarding the product information. ## Claims ### Claim 1 A system for identifying a product in an image and outputting stock keeping units of the product, comprising: a database server executed by one or more storage devices for storing image data comprising real-time inventory images and historical images of each product type; a data analytics system executed by one or more computers; and a standard dashboard; wherein the data analytics system comprises instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: receiving an input image, via application programming interface from a recording device or an external database; generating an output characterizing an object presented within the image, via an object detection component of a first neural network in the data analytics system; receiving the output generated by the first neural network, via a second neural network; processing the output, via an object classification component of the second neural network in the data analytics system, to determine hierarchical classification categories comprising a first level classification category of the object, a second level classification category of the object and a third level classification category of the object; and generating a descriptor of the object based on at least one of the first level classification category, the second level classification category and the third level classification category of the object, via the data analytics system; and wherein the standard dashboard is configured to output the descriptor of the object. ### Claim 2 (depends on 1) The system of claim 1, wherein the application programming interface is microservice based. ### Claim 3 (depends on 1) The system of claim 1, wherein the object detection component applies instance segmentation configured to extract a product at individual level and masks the product out with any other information. ### Claim 4 (depends on 3) The system of claim 3, wherein the instance segmentation is achieved based on convolutional neural network. ### Claim 5 (depends on 1) The system of claim 1, wherein the first neural network comprises a hybrid neural network model consisting of convolutional neural network and multilayer perceptron architecture. ### Claim 6 (depends on 5) The system of claim 5, wherein the first neural network comprising a convolutional layer, a pooled layer and an upsample layer. ### Claim 7 (depends on 1) The system of claim 1, wherein the second neural network comprising a convolutional layer, and a perceptron layer. ### Claim 8 (depends on 1) The system of claim 1, wherein the first and second neural networks are trained in an end-to-end automated learning process. ### Claim 9 A method for identifying a product in an image and outputting stock keeping units of the product, comprising receiving an input image, via application programming interface from a recording device or an external database; generating an output characterizing an object presented within the image, via an object detection component of a first neural network in a data analytics system; receiving the output generated by the first neural network, via a second neural network; processing the output, via an object classification component of the second neural network in the data analytics system, to determine hierarchical classification categories comprising a first level classification category of the object, a second level classification category of the object and a third level classification category of the object; generating a descriptor of the object based on at least one of the first level classification category, the second level classification category and the third level classification category of the object, via the data analytics system; and outputting the descriptor of the object, via a standard dashboard. ### Claim 10 (depends on 9) The method of claim 9, wherein the application programming interface is microservice based. ### Claim 11 (depends on 9) The method of claim 9, wherein the object detection component applies instance segmentation configured to extract a product at individual level and masks the product out with any other information. ### Claim 12 (depends on 11) The method of claim 11, wherein the instance segmentation is achieved based on convolutional neural network. ### Claim 13 (depends on 9) The method of claim 9, wherein the first neural network comprises a hybrid neural network model consisting of convolutional neural network and multilayer perceptron architecture. ### Claim 14 (depends on 13) The method of claim 13, wherein the first neural network comprising a convolutional layer, a pooled layer and an upsample layer. ### Claim 15 (depends on 9) The method of claim 9, wherein the second neural network comprising a convolutional layer, and a perceptron layer. ### Claim 16 (depends on 9) The method of claim 9, wherein the first and second neural networks are trained in an end-to-end automated learning process. ### Claim 17 A system for identifying a product in an image and outputting stock keeping units of the product, comprising: a database server executed by one or more storage devices for storing image data comprising real-time inventory images and historical images of each product type; a data analytics system executed by one or more computers; and a standard dashboard; wherein the data analytics system comprises instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: receiving an input image, via application programming interface from a recording device or an external database; generating an output characterizing an object presented within the image, via an object detection component of a first neural network in the data analytics system; generating a first descriptor for the object based on the output, via the first neural network in the data analytics system; receiving the output and the first descriptor generated by the first neural network, via a second neural network; processing the output, via an object classification component of the second neural network in the data analytics system, to determine hierarchical classification categories comprising a first level classification category of the object, a second level classification category of the object and a third level classification category of the object; and generating a second descriptor of the object based on at least one of the first level classification category, the second level classification category and the third level classification category of the object, via the data analytics system; and wherein the standard dashboard is configured to output the first and second descriptor of the object. ### Claim 18 (depends on 17) The system of claim 17, wherein the application programming interface is microservice based. ### Claim 19 (depends on 17) The system of claim 17, wherein the object detection component applies instance segmentation configured to extract a product at individual level and masks the product out with any other information. ### Claim 20 (depends on 19) The system of claim 19, wherein the instance segmentation is achieved based on convolutional neural network. ### Claim 21 (depends on 17) The system of claim 17, wherein the first neural network comprises a hybrid neural network model consisting of convolutional neural network and multilayer perceptron architecture. ### Claim 22 (depends on 21) The system of claim 21, wherein the first neural network comprising a convolutional layer, a pooled layer and an upsample layer. ### Claim 23 (depends on 17) The system of claim 17, wherein the second neural network comprising a convolutional layer, and a perceptron layer. ### Claim 24 (depends on 17) The system of claim 17, wherein the first and second neural networks are trained in an end-to-end automated learning process.