# US 2019/0166980 A1 · Systems and Methods for Identification and Virtual Application of Cosmetic Products > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0166980 A1 - Document ID: US20190166980A1 - Status: Published application - Kind: A1 - Application number: 15/909,179 - Filed: 2018-03-01 - Published: 2019-06-06 - Earliest claimed: 2017-12-01 - Assignees: Perfect Corp. - Applicants: Perfect Corp. - Inventors: Jau-Hsiung Huang; Wei-Hsin Tseng - Web page: https://patentgrep.com/patent/US20190166980A1 ## CPC - A45D 44/005: {for selecting or displaying personal cosmetic colours or hairstyle} - G06F 16/5854: {using shape and object relationship} - G06Q 30/02: Marketing; Price estimation or determination; Fundraising - G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/172: {Classification, e.g. identification} - A45D 2044/007: {Devices for determining the condition of hair or skin or for selecting the appropriate cosmetic or hair treatment} - G06T 2207/30201: Face - A45D: HAIRDRESSING OR SHAVING EQUIPMENT (wigs, toupees or the like A41G3/00, A41G5/00; hairdressers' chairs A47C1/04; hair cutting appliances, razors B26B); EQUIPMENT FOR COSMETICS OR COSMETIC TREATMENTS, e.g. FOR MANICURING OR PEDICURING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In a computing device for identifying cosmetic products and simulating application of the cosmetic products, a target image is obtained from a user, where the target image depicts at least one of a cosmetic product or an individual wearing at least one cosmetic product. The computing device accesses a database storing a plurality of sample images, each sample image having a corresponding image feature map and metadata. The computing device analyzes the target image and identifies a matching sample image among the plurality of sample images based on the image feature map. The computing device obtains an image or video with a facial region of the user via a camera and generates a user interface displaying a resulting image or video showing virtual application of the at least one cosmetic product on the user. The computing device also displays cosmetic product information to the user in the user interface. ## Claims ### Claim 1 A method implemented in a computing device for identifying cosmetic products and simulating application of the cosmetic products, comprising: obtaining a target image from a user, the target image depicting at least one of a cosmetic product or an individual wearing at least one cosmetic product; accessing a database storing a plurality of sample images, each sample image having a corresponding image feature map and metadata, the metadata comprising cosmetic product information and cosmetic makeup parameters; analyzing the target image and identifying a matching sample image among the plurality of sample images based on the image feature map; obtaining an image or video with a facial region of the user via a camera; performing virtual application of at least one cosmetic product on the image or video with the facial region of the user based on the cosmetic makeup parameters specified in metadata of the matching sample image; generating a user interface displaying a resulting image or video showing virtual application of the at least one cosmetic product on the user; and displaying cosmetic product information to the user in the user interface corresponding to the cosmetic product information specified in the metadata of the matching sample image. ### Claim 2 (depends on 1) The method of claim 1, further comprising pre-processing the target image, wherein pre-processing the target image is performed prior to analyzing the target image and identifying the matching sample image among the plurality of sample images, and wherein pre-processing of the target image comprises at least one of: a flip operation, a deskewing operation, rotation of the target image, white-balance adjustment, noise reduction, and perspective correction. ### Claim 3 (depends on 1) The method of claim 1, wherein analyzing the target image and identifying the matching sample image among the plurality of sample images comprises determining whether a threshold degree of similarity is met between a feature map of at least one cosmetic product depicted in the target image and an image feature map of a matching sample image among the plurality of sample images. ### Claim 4 (depends on 3) The method of claim 3, wherein a sample image with an image feature map with a highest degree of similarity with the at least one cosmetic product in the target image is selected as the matching sample image. ### Claim 5 (depends on 1) The method of claim 1, wherein the target image obtained from the user is captured utilizing a camera on a back of the computing device, and wherein the image or video of the facial region of the user is captured utilizing a front-facing camera of the computing device. ### Claim 6 (depends on 1) The method of claim 1, wherein the cosmetic product information of each sample image comprises at least one of: a product name, a product stock keeping unit (SKU) code for at least one cosmetic product, color number and color name associated with the at least one cosmetic product, and purchasing information for the at least one cosmetic product. ### Claim 7 (depends on 6) The method of claim 6, wherein the purchasing information for the at least one cosmetic product comprises a Uniform Resource Locator (URL) of an online retailer for a product web page selling the at least one cosmetic product. ### Claim 8 (depends on 1) The method of claim 1, wherein the cosmetic makeup parameters comprise at least one of: a color value, a make up look pattern, a transparency level, and a reflection rate specifying a matte appearance or a shiny appearance. ### Claim 9 (depends on 1) The method of claim 1, wherein the image feature map of each sample image identifies target facial features wearing at least one cosmetic product. ### Claim 10 (depends on 1) The method of claim 1, wherein the database storing the plurality of sample images is maintained by a cloud-based server. ### Claim 11 A system, comprising: a memory storing instructions; at least one camera; and a processor coupled to the memory and configured by the instructions to at least: obtain a target image from a user, the target image depicting at least one of a cosmetic product or an individual wearing at least one cosmetic product; access a database storing a plurality of sample images, each sample image having a corresponding image feature map and metadata, the metadata comprising cosmetic product information and cosmetic makeup parameters; analyze the target image and identify a matching sample image among the plurality of sample images based on the image feature map; obtain an image or video with a facial region of the user via a camera; perform virtual application of at least one cosmetic product on the image or video with the facial region of the user based on the cosmetic makeup parameters specified in metadata of the matching sample image; generate a user interface displaying a resulting image or video showing virtual application of the at least one cosmetic product on the user; and display cosmetic product information to the user in the user interface corresponding to the cosmetic product information specified in the metadata of the matching sample image. ### Claim 12 (depends on 11) The system of claim 11, wherein the processor is configured for analyzing the target image and identifying the matching sample image among the plurality of sample images by determining whether a threshold degree of similarity is met between a feature map of at least one cosmetic product depicted in the target image and an image feature map of a matching sample image among the plurality of sample images. ### Claim 13 (depends on 11) The system of claim 11, wherein the target image obtained from the user is captured utilizing a camera on a back of the system, and wherein the image of the facial region of the user is captured utilizing a front-facing camera of the system. ### Claim 14 (depends on 11) The system of claim 11, wherein the cosmetic product information of each sample image comprises at least one of: a product name, a product stock keeping unit (SKU) code for at least one cosmetic product, color number and color name associated with the at least one cosmetic product, and purchasing information for the at least one cosmetic product. ### Claim 15 (depends on 14) The system of claim 14, wherein the purchasing information for the at least one cosmetic product comprises a Uniform Resource Locator (URL) of an online retailer for a product web page selling the at least one cosmetic product. ### Claim 16 (depends on 11) The system of claim 11, wherein the image feature map of each sample image identifies target facial features wearing at least one cosmetic product. ### Claim 17 (depends on 11) The system of claim 11, wherein the database storing the plurality of sample images is maintained by a cloud-based server. ### Claim 18 A non-transitory computer-readable storage medium storing instructions to be implemented by a computing device having a processor, wherein the instructions, when executed by the processor, cause the computing device to at least: obtain a target image from a user, the target image depicting at least one of a cosmetic product or an individual wearing at least one cosmetic product; access a database storing a plurality of sample images, each sample image having a corresponding image feature map and metadata, the metadata comprising cosmetic product information and cosmetic makeup parameters; analyze the target image and identify a matching sample image among the plurality of sample images based on the image feature map; obtain an image or video with a facial region of the user via a camera; perform virtual application of at least one cosmetic product on the image or video with the facial region of the user based on the cosmetic makeup parameters specified in metadata of the matching sample image; generate a user interface displaying a resulting image or video showing virtual application of the at least one cosmetic product on the user; and display cosmetic product information to the user in the user interface corresponding to the cosmetic product information specified in the metadata of the matching sample image. ### Claim 19 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the processor is configured for analyzing the target image and identifying the matching sample image among the plurality of sample images by determining whether a threshold degree of similarity is met between a feature map of at least one cosmetic product depicted in the target image and an image feature map of a matching sample image among the plurality of sample images. ### Claim 20 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the cosmetic product information of each sample image comprises at least one of: a product name, a product stock keeping unit (SKU) code for at least one cosmetic product, color number and color name associated with the at least one cosmetic product, and purchasing information for the at least one cosmetic product. ### Claim 21 (depends on 20) The non-transitory computer-readable storage medium of claim 20, wherein the purchasing information for the at least one cosmetic product comprises a Uniform Resource Locator (URL) of an online retailer for a product web page selling the at least one cosmetic product. ### Claim 22 (depends on 18) The non-transitory computer-readable storage medium of claim 18, wherein the image feature map of each sample image identifies target facial features wearing at least one cosmetic product.