# US 12,118,601 B2 · Method, system, and non-transitory computer-readable medium for analyzing facial features for augmented reality experiences of physical products in a messaging system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,601 B2
- Document ID: US12118601B2
- Status: Granted
- Kind: B2
- Application number: 17/205,837
- Filed: 2021-03-18
- Published: 2021-10-07
- Granted: 2024-10-15
- Earliest claimed: 2020-04-01
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Jean Luo; Celia Nicole Mourkogiannis
- Web page: https://patentgrep.com/patent/US12118601B2

## CPC

- G06Q 30/0631: {Recommending goods or services}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06F 16/436: {using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06Q 30/0621: {by configuring or customising goods or services}
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06V 10/56: relating to colour
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- H04L 51/10: Multimedia information
- H04L 51/18: Commands or executable codes
- G06N 3/045: Combinations of networks
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06V 2201/10: Recognition assisted with metadata
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

The subject technology receives image data including a representation of a face of a user. The subject technology analyzes the image data to determine a set of characteristics of the representation of the face. The subject technology, based at least in part on the determined set of characteristics, selects a particular product and a set of media content associated with the particular product. The subject technology causes display, at a client device, at least one recommendation corresponding to the set of media content associated with the particular product.

## Claims

### Claim 1

A method, comprising:
receiving, by an image data receiving module of a client device, image data including a representation of a face of a user;
receiving a first selection of a portion of the representation of the face of the user;
in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user;
receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items;
analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face;
based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises:
sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product;
sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet;
performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item:
in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network:
receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and
causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user.

### Claim 2 (depends on 1)

The method of claim 1,
wherein the product metadata comprises the product identifier, a product augmented reality experience identifier, and information related to the product corresponding to the product identifier:
wherein selecting the product and the set of media content further comprises:
performing, by the product catalog system, a lookup operation on a product table to retrieve the product metadata; and
storing, by the product catalog system, the information related to the respective physical item into the product table.

### Claim 3 (depends on 1)

The method of claim 1, wherein the at least one recommendation comprises a suggested augmented content reality generator, media content, or additional textual information related to at least one characteristic from the determined set of characteristics.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determined set of characteristics includes a health-related characteristic identified in the representation of the face, and the health condition comprises at least one of dry skin, oily skin, combination skin, sunburns, acne, eczema, rosacea, ichthyosis, or dermatitis.

### Claim 5 (depends on 1)

The method of claim 1, wherein the determined set of characteristics includes a shape of the face, or a skin tone of the face.

### Claim 6 (depends on 1)

The method of claim 1, wherein analyzing the image data to determine the set of characteristics of the representation of the face further comprises:
performing a facial feature detection process to detect a first facial feature; and
identifying a health-related characteristic associated with the detected first facial feature.

### Claim 7 (depends on 6)

The method of claim 6, wherein the facial feature detection process comprises a machine learning algorithm.

### Claim 8 (depends on 7)

The method of claim 7, wherein the machine learning algorithm is based on a neural network model.

### Claim 9 (depends on 6)

The method of claim 6, further comprising:
performing a second facial feature detection process to detect the first facial feature, the second facial feature detection process being different than the facial feature detection process.

### Claim 10 (depends on 1)

The method of claim 1, wherein first selection of the portion of the representation of the face of the user corresponds to an eye of the face of the user, and
wherein the determined set of characteristics comprises different types of eyelashes, the different type of eyelashes comprising dry eyelashes and weak eyelashes,
wherein the at least one recommendation comprises textual information related to addressing dry eyelashes.

### Claim 11

A system comprising:
a processor; and
a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving, by an image data receiving module of a client device, image data including a representation of a face of a user;
receiving a first selection of a portion of the representation of the face of the user;
in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user;
receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items;
analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face;
based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises:
sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product;
sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet:
performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item:
in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network:
receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and
causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user.

### Claim 12 (depends on 11)

The system of claim 11,
wherein the product metadata comprises the product identifier, a product augmented reality experience identifier, and information related to the product corresponding to the product identifier:
performing, by the product catalog system, a lookup operation on a product table to retrieve product metadata; and
storing, by the product catalog system, the information related to the respective physical item into the product table.

### Claim 13 (depends on 11)

The system of claim 11, wherein the at least one recommendation comprises a suggested augmented content reality generator, media content, or additional textual information related to at least one characteristic from the determined set of characteristics.

### Claim 14 (depends on 11)

The system of claim 11, wherein the determined set of characteristics includes a health-related characteristic identified in the representation of the face.

### Claim 15 (depends on 11)

The system of claim 11, wherein the determined set of characteristics includes a shape of the face, or a skin tone of the face.

### Claim 16 (depends on 11)

The system of claim 11, wherein analyzing the image data to determine the set of characteristics of the representation of the face further causes the processor to perform further operations comprising:
performing a facial feature detection process to detect a first facial feature; and
identifying a health-related characteristic associated with the detected first facial feature.

### Claim 17 (depends on 16)

The system of claim 16, wherein the facial feature detection process comprises a machine learning algorithm.

### Claim 18 (depends on 17)

The system of claim 17, wherein the machine learning algorithm is based on a neural network model.

### Claim 19 (depends on 16)

The system of claim 16, wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:
performing a second facial feature detection process to detect the first facial feature, the second facial feature detection process being different than the facial feature detection process.

### Claim 20

A non-transitory computer-readable medium comprising instructions,
which when executed by a computing device, cause the computing device to perform operations comprising:
receiving, by an image data receiving module of a client device, image data including a representation of a face of a user;
receiving a first selection of a portion of the representation of the face of the user;
in response to the first selection of the portion of the representation of the face of the user, causing display of a plurality of selectable graphical items, each of the plurality of selectable graphical items corresponding to a particular state of the portion of the representation of the face of the user;
receiving a second selection of a particular selectable graphical item from the plurality of selectable graphical items;
analyzing, by a product identification module accessing a shared memory of the client device, the image data to determine a set of characteristics of the representation of the face;
based at least in part on the determined set of characteristics and the second selection of the particular selectable graphical item from the plurality of selectable graphical items, selecting a product and a set of media content associated with the product, the product comprising a particular product that, when utilized, addresses a facial condition included in the determined set of characteristics, wherein selecting the product and the set of media content comprises:
sending, by the client device to a product catalog system over the Internet, an indicator of the selected product, the product catalog system comprising a first processor for processing product metadata and a first memory for storing information related to second product metadata associated with the product metadata of the selected product, the information comprising at least a set of manufacturers of the selected product;
sending, by the product catalog system, a request message over the Internet to a product catalog service system for obtaining metadata related to a respective physical item associated with the selected product, the request message including a product identifier, and the product catalog system and the product catalog service system being different systems that are communicatively coupled over the Internet:
performing, by the product catalog service system, a query on a database over a network based on the product identifier in the request message for information related to the respective physical item:
in response to the request message, providing, by the product catalog service system, a response message to the product catalog system, the response message including the second product metadata based on the information related to the respective physical item from the database over the network;
receiving, by the client device from the product catalog system over the Internet, the second product metadata, the second product metadata including additional information related to the selected product, the additional information comprising information that the selected product addresses the facial condition; and
causing display, at the client device, of at least one recommendation corresponding to the set of media content associated with the product, the at least one recommendation facilitating identifying a set of products, including the product, that address at least one health condition, corresponding to the facial condition, through using the set of products by the user.