# US 12,118,751 B2 · Method for identifying a lip-makeup product appearing in an image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,751 B2 - Document ID: US12118751B2 - Status: Granted - Kind: B2 - Application number: 17/545,536 - Filed: 2021-12-08 - Published: 2022-06-09 - Granted: 2024-10-15 - Earliest claimed: 2020-12-09 - Assignees: Chanel Parfums Beaute - Applicants: Chanel Parfums Beaute - Inventors: Jonathan Loriette - Web page: https://patentgrep.com/patent/US12118751B2 ## CPC - A45D 44/005: {for selecting or displaying personal cosmetic colours or hairstyle} - G06T 7/90: Determination of colour characteristics - G06F 16/90335: {Query processing} - G06N 20/00: Machine learning - G06Q 30/0251: Targeted advertisements - G06Q 30/0601: Electronic shopping [e-shopping] - G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529) - G06V 10/74: Image or video pattern matching; Proximity measures in feature spaces - G06V 10/763: {Non-hierarchical techniques, e.g. based on statistics of modelling distributions} - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 20/20: in augmented reality scenes - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/172: {Classification, e.g. identification} - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/30201: Face - A45D: HAIRDRESSING OR SHAVING EQUIPMENT (wigs, toupees or the like A41G3/00, A41G5/00; hairdressers' chairs A47C1/04; hair cutting appliances, razors B26B); EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Disclosed is a method for identifying a lip-makeup product from an image, the method including: detecting a mouth on the image; determining whether the mouth is made-up or not; and if the mouth of the person is made-up: determining color parameters of the lip-makeup of the image from at least some of the pixels of the image belonging to the mouth; computing a distance between the color parameters of the lip-makeup from the image and the color parameters of at least some of a plurality of reference lip-makeup products; and if at least one distance between the color parameters of the lip-makeup of the input image and one of the reference lip-makeup product is below a predetermined threshold, identifying a reference lip-makeup product best matching the lip-makeup product on the image. ## Claims ### Claim 1 A method for identifying a lip-makeup product from an input image, the method being implemented by a system comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, and comprising: detecting a mouth of a person on the input image, determining whether the mouth is made-up or not, and if the mouth of the person is made-up, determining color parameters of the lip-makeup of the input image from at least some of the pixels of the image belonging to the mouth, computing a distance between the color parameters of the lip-makeup from the image and the color parameters of at least some of the plurality reference lip-makeup products, and if at least one distance between the color parameters of the lip-makeup of the input image and one of the reference lip-makeup product is below a predetermined threshold, identifying a reference lip-makeup product best matching the lip-makeup product on the image, wherein identifying a reference lip-makeup product best matching the lip-makeup product on the image comprises implementing a classification model trained on a learning dataset comprising, for each reference lip-makeup product, a plurality of artificially generated images of faces wearing said reference lip-makeup product. ### Claim 2 (depends on 1) The lip-makeup identification method according to claim 1, wherein identifying a reference lip makeup product best matching the lip-makeup product on the image comprises implementing a classification model configured to receive as input a mouth image and to output a selected one of all the reference lip-makeup products. ### Claim 3 (depends on 1) The lip-makeup identification method according to claim 1, wherein determining whether the mouth is made-up or not comprises implementing a binary classification model trained on a plurality of mouth images and configured to determine presence or absence of lip-makeup product. ### Claim 4 (depends on 1) The lip-makeup identification method according to claim 1, wherein determining color parameters of the lip-makeup of the input image comprises: selecting a set of pixels of the input image belonging to the mouth, removing black and white pixels from the set of pixels, implementing a k-means algorithm with a single cluster over the color parameters of the set of pixels, and determining the color parameters of the lip-makeup as the centroid of the color parameters of the set of pixels. ### Claim 5 (depends on 1) The lip-makeup identification method according to claim 1, wherein the color parameters of the reference lip-makeup products are obtained, for each reference lip-makeup product, by: performing virtual try-on of said lip-makeup reference product on a set of faces comprising a plurality of face images, for each obtained image, selecting a set of pixels belonging to the mouth, removing black and white pixels from the set of pixels, implementing a k-means algorithm with a single cluster over the color parameters of the set of pixels, thereby outputting a centroid, and determining the color parameters of the reference lip makeup product as a set comprising the color parameters of the centroid of each obtained image, and computing a distance between the color parameters of the lip-makeup from the image and the color parameters of at least some of the plurality of reference lip-makeup products, comprises, for each reference lip-makeup product for which a distance is computed, computing a distance between the color parameters of the lip-makeup product from the image and the color parameters of each centroid obtained for each obtained image. ### Claim 6 (depends on 1) The lip-makeup identification method according to claim 1, further comprising a step of color correction of the input image prior to determining the color parameters of the lip-makeup product of the input image. ### Claim 7 (depends on 1) The lip-makeup identification method according to claim 1, comprising a preliminary step of receiving the input image from a user terminal, and further comprising, once a reference lip-makeup product corresponding to the lip-makeup product of the image is determined, sending a return message to the user terminal comprising an identification of the reference lip-makeup product. ### Claim 8 (depends on 7) The lip-makeup identification method according to claim 7, wherein the return message further comprises connection information suitable for accessing an e-shop selling said reference lip-makeup product, and/or a virtual try-on application for virtually trying said reference lip-makeup product. ### Claim 9 (depends on 1) A lip-makeup identification system, comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, the lip-makeup identification system being configured to be capable of implementing the method according to claim 1. ### Claim 10 (depends on 1) A method for requesting identification about a lip-makeup product, implemented by a user terminal comprising a computer and a camera, and a lip-makeup-identification system comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, said method comprising: the user terminal acquiring an image, the user terminal transmitting to the lip-makeup identification system the acquired image and a request for identifying a lip-makeup product appearing on the acquired image, the lip-makeup identification system implementing on said image the method according to claim 1, and the user terminal receiving from the lip-makeup identification system, in response to said request, one among the following: an indication of the lip-makeup product appearing on the image indication that no lip-makeup product has been found in the image, or indication that no corresponding lip-makeup product has been identified. ### Claim 11 (depends on 1) A non-transitory computer-readable medium on which is stored a computer program comprising code instructions which, when executed by a computer, cause the computer to perform the method according to claim 1. ### Claim 12 A method for identifying a lip-makeup product from an input image, the method being implemented by a system comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, and comprising: detecting a mouth of a person on the input image, determining whether the mouth is made-up or not, and if the mouth of the person is made-up, determining color parameters of the lip-makeup of the input image from at least some of the pixels of the image belonging to the mouth, computing a distance between the color parameters of the lip-makeup from the image and the color parameters of at least some of the plurality of reference lip-makeup products, and if at least one distance between the color parameters of the lip-makeup of the input image and one of the reference lip-makeup product is below a predetermined threshold, identifying a reference lip-makeup product best matching the lip-makeup product on the image, further comprising determining a texture of the lip-makeup product of the image, and the step of computing a distance between the color parameters of the lip-makeup product of the image and the color parameters of the referenced lip-makeup products is performed only for the reference lip-makeup products having the same texture as the lip-makeup product of the image. ### Claim 13 (depends on 12) The lip-makeup identification method according to claim 12, wherein the steps of determining a texture of the lip-makeup product and determining whether the mouth on the image is made-up or not are performed simultaneously by implementing a trained model configured to output, from an input image representing a face, a category selected among a plurality of categories including absence of lip-makeup product and a plurality of textures of lip-makeup product. ### Claim 14 (depends on 12) A lip-makeup identification system, comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, the lip-makeup identification system being configured to be capable of implementing the method according to claim 12. ### Claim 15 A method for identifying a lip-makeup product from an input image, the method being implemented by a system comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, and comprising: detecting a mouth of a person on the input image, determining whether the mouth is made-up or not, and if the mouth of the person is made-up, determining color parameters of the lip-makeup of the input image from at least some of the pixels of the image belonging to the mouth, a distance between the color parameters of the lip-makeup from the image and the color parameters of at least some of the plurality of reference lip-makeup products, and if at least one distance between the color parameters of the lip-makeup of the input image and one of the reference lip-makeup product is below a predetermined threshold, identifying a reference lip-makeup product best matching the lip-makeup product on the image, implemented by a system further comprising an advertisements database comprising a plurality of advertisement images associated with respective reference lip-makeup products, wherein the method comprises a preliminary step of determining whether the input image corresponds to an advertisement image contained in the advertisement database and, if so, identifying the lip-makeup product on the input image as the reference lip-makeup product associated to said advertisement image, wherein determining whether the input image corresponds to an advertisement database is performed by implementing a machine learning algorithm trained on a learning database comprising the advertisement images of the advertisement database and, for each advertisement image, a set of additional images artificially generated from the advertisement image by implementing at least one transformation of said advertisement image. ### Claim 16 (depends on 15) The lip-makeup identification method according to claim 15, wherein the transformations performed on each advertisement image for generating additional images are one or a combination of: geometric deformations, modified background layer, screen-like distorsion, addition of shadow or sun-flare, color deviation. ### Claim 17 (depends on 15) A lip-makeup identification system, comprising a computer and a memory storing, for each of a plurality of reference lip-makeup products, an identifier and color parameters, the lip-makeup identification system being configured to be capable of implementing the method according to claim 15.