# US 10,769,198 B1 · Systems and methods for product identification using image analysis from image mask and trained neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,198 B1 - Document ID: US10769198B1 - Status: Granted - Kind: B1 - Application number: 16/783,289 - Filed: 2020-02-06 - Granted: 2020-09-08 - Earliest claimed: 2020-02-06 - Assignees: CaaStle, Inc. - Applicants: CaaStle, Inc. - Inventors: Yu-Cheng Tsai; Dongming Jiang; Georgiy Goldenberg - Web page: https://patentgrep.com/patent/US10769198B1 ## CPC - G06Q 30/0625: {by formulating product or service queries, e.g. using keywords or predefined options} - G06F 16/55: Clustering; Classification - G06F 16/583: using metadata automatically derived from the content - G06F 16/5854: {using shape and object relationship} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/2431: Multiple classes - G06T 7/11: Region-based segmentation - G06T 7/194: involving foreground-background segmentation - G06V 10/764: using classification, e.g. of video objects - G06V 10/82: using neural networks - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20088: Trinocular vision calculations; trifocal tensor - G06T 2207/30124: Fabrics; Textile; Paper - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed are methods, systems, and non-transitory computer-readable medium for analysis of images including wearable items. For example, a method may include obtaining a first set of images, each of the first set of images depicting a product; obtaining a first set of labels associated with the first set of images; training an image segmentation neural network based on the first set of images and the first set of labels; obtaining a second set of images, each of the second set of images depicting a known product; obtaining a second set of labels associated with the second set of images; training an image classification neural network based on the second set of images and the second set of labels; receiving a query image depicting a product that is not yet identified; and performing image segmentation of the query image and identifying the product in the image by performing image analysis. ## Claims ### Claim 1 A computer-implemented method comprising: obtaining, by one or more processors, a first set of images, wherein each of the first set of images depicts a product; obtaining, by the one or more processors, a first set of labels associated with the first set of images, wherein each of the first set of labels corresponds to an image in the first set of images and includes information indicating a mask of the product depicted in the corresponding image; training, by the one or more processors, an image segmentation neural network based on the first set of images and the first set of labels; obtaining, by the one or more processors, a second set of images, wherein each of the second set of images depicts a known product; obtaining, by the one or more processors, a second set of labels associated with the second set of images, wherein each of the second set of labels corresponds to an image in the second set of images and includes information indicating classification information for the known product depicted in the corresponding image; training, by the one or more processors, an image classification neural network based on the second set of predetermined images and the second set of labels; receiving, by the one or more processors, a query image depicting a product that is not yet identified; performing, by the one or more processors, image segmentation of the query image using the image segmentation neural network, wherein performing image segmentation of the query image includes removing background image portions of the query image, thereby obtaining a mask image of the product depicted in the query image; and identifying the product in the image by performing, by the one or more processors, image analysis of the mask image of the product depicted in the query image using the image classification neural network. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the classification information comprises one or more categories of the product, one or more patterns of the product, one or more colors of the product, one or more sleeve lengths of the product, one or more hemline lengths of the product, one or more neckline shapes of the product, one or more pattern spatial frequencies associated with the product, one or more hues associated the product, one or more saturations associated with the product, and/or one or more lightness values associated with the product. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising: testing, by the one or more processors, the image classification neural network based on the second set of predetermined images and the second set of labels. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein performing image analysis of the mask image comprises: classifying the product depicted in the query image, wherein classifying the product depicted in the query image includes determining one or more categories of the depicted product, determining one or more patterns of the depicted product, determining one or more colors of the depicted product, determining one or more sleeve lengths of the depicted product, determining one or more hemline lengths of the depicted product, determining one or more neckline shapes of the depicted product, determining one or more pattern spatial frequencies associated with the depicted product, determining one or more hues associated the depicted product, determining one or more saturations associated with the depicted product, and/or determining one or more lightness associated with the depicted product. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising: generating one or more recommendations based on the identified product, wherein the one or more recommendations include one or more products associated with the classified product depicted in the query image. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein generating the one or more recommendations based on the identified product comprises: obtaining the one or more products based on the classified product depicted in the query image; determining a similarity score for each of the one or more products; and ranking the one or more products based on the similarity scores. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, further comprising displaying one or more of the ranked products. ### Claim 8 A computer system comprising: a data storage device storing processor-readable instructions; and a processor configured to execute the instructions to perform a method including: obtaining a first set of images, wherein each of the first set of images depicts a product; obtaining a first set of labels associated with the first set of images, wherein each of the first set of labels corresponds to an image in the first set of images and includes information indicating a mask of the product depicted in the corresponding image; training an image segmentation neural network based on the first set of images and the first set of labels; obtaining a second set of images, wherein each of the second set of images depicts a known product; obtaining a second set of labels associated with the second set of images, wherein each of the second set of labels corresponds to an image in the second set of images and includes information indicating classification information for the known product depicted in the corresponding image; training an image classification neural network based on the second set of predetermined images and the second set of labels; receiving a query image depicting a product that is not yet identified; performing image segmentation of the query image using the image segmentation neural network, wherein performing image segmentation of the query image includes removing background image portions of the query image, thereby obtaining a mask image of the product depicted in the query image; and identifying the product in the image by performing image analysis of the mask image of the product depicted in the query image using the image classification neural network. ### Claim 9 (depends on 8) The computer system of claim 8, wherein the classification information comprises one or more categories of the product, one or more patterns of the product, one or more colors of the product, one or more sleeve lengths of the product, one or more hemline lengths of the product, one or more neckline shapes of the product, one or more pattern spatial frequencies associated with the product, one or more hues associated the product, one or more saturations associated with the product, and/or one or more lightness values associated with the product. ### Claim 10 (depends on 8) The computer system of claim 8, wherein the method further includes: testing the image classification neural network based on the second set of predetermined images and the second set of labels. ### Claim 11 (depends on 8) The computer system of claim 8, wherein performing image analysis of the mask image comprises: classifying the product depicted in the query image, wherein classifying the product depicted in the query image includes determining one or more categories of the depicted product, determining one or more patterns of the depicted product, determining one or more colors of the depicted product, determining one or more sleeve lengths of the depicted product, determining one or more hemline lengths of the depicted product, determining one or more neckline shapes of the depicted product, determining one or more pattern spatial frequencies associated with the depicted product, determining one or more hues associated the depicted product, determining one or more saturations associated with the depicted product, and/or determining one or more lightness values associated with the depicted product. ### Claim 12 (depends on 11) The computer system of claim 11, the method further including: generating one or more recommendations based on the identified product, wherein the one or more recommendations include one or more products associated with the classified product depicted in the query image. ### Claim 13 (depends on 12) The computer system of claim 12, wherein generating the one or more recommendations based on the identified product comprises: obtaining the one or more products based on the classified product depicted in the query image; determining a similarity score for each of the one or more products; and ranking the one or more products based on the similarity scores. ### Claim 14 (depends on 13) The computer system of claim 13, further comprising displaying one or more of the ranked products. ### Claim 15 A non-transitory computer-readable medium containing instructions that, when executed by a processor, cause the processor to perform a method comprising: obtaining a first set of images, wherein each of the first set of images depicts a product; obtaining a first set of labels associated with the first set of images, wherein each of the first set of labels corresponds to an image in the first set of images and includes information indicating a mask of the product depicted in the corresponding image; training an image segmentation neural network based on the first set of images and the first set of labels; obtaining a second set of images, wherein each of the second set of images depicts a known product; obtaining a second set of labels associated with the second set of images, wherein each of the second set of labels corresponds to an image in the second set of images and includes information indicating classification information for the known product depicted in the corresponding image; training an image classification neural network based on the second set of predetermined images and the second set of labels; receiving a query image depicting a product that is not yet identified; performing image segmentation of the query image using the image segmentation neural network, wherein performing image segmentation of the query image includes removing background image portions of the query image, thereby obtaining a mask image of the product depicted in the query image; and identifying the product in the image by performing image analysis of the mask image of the product depicted in the query image using the image classification neural network. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the classification information comprises one or more categories of the product, one or more patterns of the product, one or more colors of the product, one or more sleeve lengths of the product, one or more hemline lengths of the product, one or more neckline shapes of the product, one or more pattern spatial frequencies associated with the product, one or more hues associated the product, one or more saturations associated with the product, and/or one or more lightness values associated with the product. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the method further includes: testing the image classification neural network based on the second set of predetermined images and the second set of labels. ### Claim 18 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein performing image analysis of the mask image comprises: classifying the product depicted in the query image, wherein classifying the product depicted in the query image includes determining one or more categories of the depicted product, determining one or more patterns of the depicted product, determining one or more colors of the depicted product, determining one or more sleeve lengths of the depicted product, determining one or more hemline lengths of the depicted product, determining one or more neckline shapes of the depicted product, determining one or more pattern spatial frequencies associated with the depicted product, determining one or more hues associated the depicted product, determining one or more saturations associated with the depicted product, and/or determining one or more lightness values associated with the depicted product. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, the method further comprising: generating one or more recommendations based on the identified product, wherein the one or more recommendations include one or more products associated with the classified product depicted in the query image. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein generating the one or more recommendations based on the identified product comprises: obtaining the one or more products based on the classified product depicted in the query image; determining a similarity score for each of the one or more products; ranking the one or more products based on the similarity scores; and displaying one or more of the ranked products.