# US 2019/0318405 A1 · Product identification in image with multiple products > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318405 A1 - Document ID: US20190318405A1 - Status: Published application - Kind: A1 - Application number: 15/954,152 - Filed: 2018-04-16 - Published: 2019-10-17 - Earliest claimed: 2018-04-16 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing , LLC - Inventors: Houdong Hu; Li Huang - Web page: https://patentgrep.com/patent/US20190318405A1 ## CPC - G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 16/53: Querying - G06F 16/532: Query formulation, e.g. graphical querying - G06F 16/55: Clustering; Classification - G06F 18/24: Classification techniques - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06V 10/82: using neural networks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 30/19173: {Classification techniques} - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/048: Activation functions - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06N 5/041: {Abduction} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N 20/20: Ensemble learning - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Methods, systems, and computer programs are presented for identifying the brand and model of products embedded within an image. One method includes operations for receiving, via a graphical user interface (GUI), a selection of an image, and for analyzing the image to determine a location within the image of one or more products. For each product in the image, determining a unique identification of the product is determined, the unique identification including a manufacturer of the product and a model identifier. The method further includes an operation for presenting information about the one or more products in the GUI with a selection option for selecting each of the one or more products. Additionally, the method includes operations for receiving a product selection for one of the one or more products, and presenting shopping options in the GUI for purchasing the selected product. ## Claims ### Claim 1 A method comprising: receiving, via a graphical user interface (GUI), a selection of an image; analyzing, by one or more processors, the image to determine a location within the image of one or more products; for each product in the image, determining, by the one or more processors, a unique identification of the product that includes a manufacturer of the product and a model identifier; presenting, by the one or more processors, information about the one or more products in the GUI with a selection option for selecting each of the one or more products; receiving, by the one or more processors, a product selection for one of the one or more products; and presenting, by the one or more processors, shopping options in the GUI for purchasing the selected product. ### Claim 2 (depends on 1) The method as recited in claim 1, wherein determining the unique identification of the product further includes: utilizing a machine-learning classifier for visually recognizing image data associated with the product. ### Claim 3 (depends on 2) The method as recited in claim 2, wherein the machine-learning classifier is trained with labeled data, the labeled data including one or more of information from a vendor's database, a product database, or information gathered from a shopping website. ### Claim 4 (depends on 1) The method as recited in claim 1, wherein determining the unique identification of the product further includes: utilizing a first neural network classifier to determine a category of the product; and utilizing a second neural network classifier with bi-linear pooling for visually recognizing image data associated with the product based on the category, the bi-linear pooling including two feature extractors whose outputs are combined to obtain an image descriptor. ### Claim 5 (depends on 1) The method as recited in claim 1, wherein determining the unique identification of the product further includes: performing an image search of image data associated with the product, the image search producing a plurality of results; and determining the unique identification of the product based on a plurality of top results from the plurality of results. ### Claim 6 (depends on 5) The method as recited in claim 5, wherein determining the unique identification of the product based on a plurality of top results further includes: determining the plurality of top results as a predetermined number of results having a highest probability of being a match for the product; and using the plurality of top results as votes to determine a product with a highest number of matches within the top results. ### Claim 7 (depends on 1) The method as recited in claim 1, wherein analyzing the image to determine the location within the image of the one or more products further includes: identifying a bounding box within the image for each of the one or more products. ### Claim 8 (depends on 1) The method as recited in claim 1, wherein presenting information about the one or more products in the GUI further includes: presenting a model number next to each of the one or more products; and presenting, for each of the one or more products, a bounding box within the image. ### Claim 9 (depends on 1) The method as recited in claim 1, wherein analyzing the image to determine the location within the image of one or more products further includes: training a machine-learning program to generate feature maps for feature extraction, object detection, and object classification. ### Claim 10 (depends on 1) The method as recited in claim 1, wherein presenting the shopping options further includes: presenting, for each shopping option, one or more of a shopping website link, a shopping website link, and a price for the product. ### Claim 11 A system comprising: a memory comprising instructions; and one or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the one or more computer processors to perform operations comprising: receiving, via a graphical user interface (GUI), a selection of an image; analyzing the image to determine a location within the image of one or more products; for each product in the image, determining a unique identification of the product that includes a manufacturer of the product and a model identifier; presenting information about the one or more products in the GUI with a selection option for selecting each of the one or more products; receiving a product selection for one of the one or more products; and presenting shopping options in the GUI for purchasing the selected product. ### Claim 12 (depends on 11) The system as recited in claim 11, wherein determining the unique identification of the product further includes: utilizing a machine-learning classifier for visually recognizing image data associated with the product, wherein the machine-learning classifier is trained with labeled data, the labeled data including one or more of information from a vendor's database, a product database, or information gathered from a shopping website. ### Claim 13 (depends on 11) The system as recited in claim 11, wherein determining the unique identification of the product further includes: utilizing neural network classifiers with bi-linear pooling for visually recognizing image data associated with the product, the bi-linear pooling including two feature extractors whose outputs are combined to obtain an image descriptor. ### Claim 14 (depends on 11) The system as recited in claim 11, wherein determining the unique identification of the product further includes: performing an image search of image data associated with the product, the image search producing a plurality of results; and determining the unique identification of the product based on a plurality of top results from the plurality of results. ### Claim 15 (depends on 14) The system as recited in claim 14, wherein determining the unique identification of the product based on a plurality of top results further includes: determining the plurality of top results as a predetermined number of results having a highest probability of being a match for the product; and using the plurality of top results as votes to determine a product with a highest number of matches within the top results. ### Claim 16 A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising: receiving, via a graphical user interface (GUI), a selection of an image; analyzing the image to determine a location within the image of one or more products; for each product in the image, determining a unique identification of the product that includes a manufacturer of the product and a model identifier; presenting information about the one or more products in the GUI with a selection option for selecting each of the one or more products; receiving a product selection for one of the one or more products; and presenting shopping options in the GUI for purchasing the selected product. ### Claim 17 (depends on 16) The non-transitory machine-readable storage medium as recited in claim 16, wherein determining the unique identification of the product further includes: utilizing a machine-learning classifier for visually recognizing image data associated with the product, wherein the machine-learning classifier is trained with labeled data, the labeled data including one or more of information from a vendor's database, a product database, or information gathered from a shopping website. ### Claim 18 (depends on 16) The non-transitory machine-readable storage medium as recited in claim 16, wherein determining the unique identification of the product further includes: utilizing neural network classifiers with bi-linear pooling for visually recognizing image data associated with the product, the bi-linear pooling including two feature extractors whose outputs are combined to obtain an image descriptor. ### Claim 19 (depends on 16) The non-transitory machine-readable storage medium as recited in claim 16, wherein determining the unique identification of the product further includes: performing an image search of image data associated with the product, the image search producing a plurality of results; and determining the unique identification of the product based on a plurality of top results from the plurality of results. ### Claim 20 (depends on 19) The non-transitory machine-readable storage medium as recited in claim 19, wherein determining the unique identification of the product based on a plurality of top results further includes: determining the plurality of top results as a predetermined number of results having a highest probability of being a match for the product; and using the plurality of top results as votes to determine a product with a highest number of matches within the top results.