# US 12,112,573 B2 · Asymmetric facial expression recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,573 B2 - Document ID: US12112573B2 - Status: Granted - Kind: B2 - Application number: 17/402,344 - Filed: 2021-08-13 - Published: 2023-02-16 - Granted: 2024-10-08 - Earliest claimed: 2021-08-13 - Assignees: Lemon Inc. - Applicants: Lemon Inc. - Inventors: Michael Leong Hou Tay; Wanchun Ma; Shuo Cheng; Chao Wang; Linjie Luo - Web page: https://patentgrep.com/patent/US12112573B2 ## CPC - G06V 40/176: {Dynamic expression} - G06F 18/2193: {based on specific statistical tests} - G06T 7/251: {involving models} - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites - G06V 10/242: {by image rotation, e.g. by 90 degrees} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure describes techniques for facial expression recognition. A first loss function may be determined based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions. A second loss function may be determined based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions. The first loss function and the second loss function may be used to determine a maximum loss function. The maximum loss function may be applied during training of a model. The trained model may be configured to predict at least one asymmetric facial expression in a subsequently received image. ## Claims ### Claim 1 A method, comprising: determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions; determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions; determining, based on the first loss function and the second loss function, a maximum loss function; applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and controlling a facial animation using the predicted different types of asymmetric facial expressions. ### Claim 2 (depends on 1) The method of claim 1, wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity. ### Claim 3 (depends on 1) The method of claim 1, wherein the second set of labels comprises blend shape coefficients indicative of the asymmetric facial expressions associated with at least one of an eye, a mouth, a jaw, an eyebrow, a cheek, a nose, or a tongue. ### Claim 4 (depends on 1) The method of claim 1, wherein the second set of images correspond to a subset of the first set of images. ### Claim 5 (depends on 1) The method of claim 1, further comprising: augmenting the second set of images by flipping at least one portion of at least one image of the second set of images. ### Claim 6 (depends on 5) The method of claim 5, further comprising: augmenting the second set of labels based on at least one flipped portion of at least one image. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images. ### Claim 8 A system, comprising: at least one computing device in communication with a computer memory, the computer memory comprising computer-readable instructions that upon execution by the at least one computing device, configure the system to perform operations comprising: determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions; determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions; determining, based on the first loss function and the second loss function, a maximum loss function; applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and controlling a facial animation using the predicted different types of asymmetric facial expressions. ### Claim 9 (depends on 8) The system of claim 8, wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity. ### Claim 10 (depends on 8) The system of claim 8, wherein the second set of labels comprises blend shape coefficients indicative of the asymmetric facial expressions associated with at least one of an eye, a mouth, a jaw, an eyebrow, a cheek, a nose, or a tongue. ### Claim 11 (depends on 8) The system of claim 8, wherein the second set of images correspond to a subset of the first set of images. ### Claim 12 (depends on 8) The system of claim 8, the operations further comprising: augmenting the second set of images by flipping at least one portion of at least one image of the second set of images. ### Claim 13 (depends on 12) The system of claim 12, the operations further comprising: augmenting the second set of labels based on at least one flipped portion of at least one image. ### Claim 14 (depends on 8) The system of claim 8, the operations further comprising: determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images. ### Claim 15 A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising: determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions; determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions; determining, based on the first loss function and the second loss function, a maximum loss function; applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and controlling a facial animation using the predicted different types of asymmetric facial expressions. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity. ### Claim 17 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the operations further comprising: determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images.