# US 2026/0073608 A1 · Training instances of machine learning model for facial expression prediction and generating new avatars used in training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0073608 A1
- Document ID: US20260073608A1
- Status: Published application
- Kind: A1
- Application number: 19/106,170
- Filed: 2022-09-02
- Published: 2026-03-12
- Earliest claimed: 2022-09-02
- Assignees: Hewlett-Packard Development Company, L.P.; Purdue Research Foundation
- Applicants: Hewlett-Packard Development Company, L.P.; Purdue Research Foundation
- Inventors: Yingying Huang; Xiaoyu Ji; Jishang Wei; Justin Yang; Prahalathan Sundaramoorthy; Shibo Zhang; Fengqing Zhu; Jan Philip Allebach; Qian Lin
- Web page: https://patentgrep.com/patent/US20260073608A1

## CPC

- G06V 10/776: Validation; Performance evaluation
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/20: in augmented reality scenes
- G06V 40/175: {Static expression}
- G06V 40/176: {Dynamic expression}
- G06T 2219/2021: Shape modification
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

For each avatar, testing images are rendered for different facial expressions that each have ground truth facial action units. An instance of a machine learning model is applied to the testing images to generate predicted facial action units for each testing image. A predictive performance of the instance is calculated for each avatar based on the predicted and ground truth facial action units for the testing images of the avatar. A first set of features common to the avatars for which the predictive performance was better than a first threshold, and a second set of features common to the avatars for which the predictive performance was worse than a second threshold, are identified. The features present only in the second set are identified, as difference features. New avatars having the difference features are generated. WO

## Claims

### Claim 1

A non-transitory computer-readable medium storing program code executable by a processor to perform processing comprising:
for each of a plurality of avatars, rendering images for different facial expressions that each have ground truth facial action units;
applying a machine learning model to the images to generate predicted facial action units for each image;
calculating a predictive performance of the machine learning model for each avatar based on the predicted and ground truth facial action units for the images of the avatar;
identifying a first set of features common to the avatars for which the predictive performance was better than a first threshold;
identifying a second set of features common to the avatars for which the predictive performance was worse than a second threshold;
identifying the features present only in the second set, as difference features; and
generating new avatars having the difference features.

### Claim 2 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the processing further comprises:
retraining the machine learning model using the new avatars.

### Claim 3 (depends on 2)

The non-transitory computer-readable medium of claim 2, wherein the processing further comprises:
applying the retrained machine learning model to facial images captured by a head-mountable display (HMD) of a wearer exhibiting a facial expression to generate predicted wearer facial action units for the facial expression of the wearer;
retargeting the predicted wearer facial action units onto an avatar corresponding to the wearer to render the avatar with the facial expression of the wearer; and
displaying the rendered avatar corresponding to the wearer.

### Claim 4 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein rendering the images comprises:
for each different facial expression, rendering a corresponding image for each avatar.

### Claim 5 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein calculating the predictive performance of the machine learning model for each avatar comprises:
calculating a mean absolute error or a mean square error between the predicted facial action units and the ground truth facial action units for the images of the avatar.

### Claim 6 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the first threshold comprises a highest quartile of the predictive performance over all the avatars,
and wherein the second threshold comprises a lowest quartile of the predictive performance over all the avatars.

### Claim 7 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the features of the avatars comprise facial geometry features.

### Claim 8

A method comprising:
selecting training avatars and testing avatars from a plurality of avatars;
for each training avatar, rendering a plurality of training images for different facial expressions that each correspond to ground truth facial action units;
training a plurality of instances of a machine learning model using the training images, each instance corresponding to different training parameters;
for each testing avatar, rendering a plurality of testing images for the different facial expressions;
applying each instance of the machine learning model to the testing images to generate predicted facial action units for each testing image; and
calculating a predictive performance of each instance of the machine learning model based on the predicted and ground truth facial action units for the testing images of the testing avatars.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
applying the instance of the machine learning model having the predictive performance that is best to facial images captured by a head-mountable display (HMD) of a wearer exhibiting a facial expression to generate predicted wearer facial action units of the wearer;
retargeting the predicted wearer facial action units onto an avatar corresponding to the wearer to render the avatar with the facial expression of the wearer; and
displaying the rendered avatar corresponding to the wearer.

### Claim 10 (depends on 8)

The method of claim 8, wherein calculating the predictive performance of each instance of the machine learning model comprises:
calculating a mean absolute error between the predicted facial action units generated by the instance and the ground truth facial action units for the testing images of the testing avatars.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
applying the instance of the machine learning model having the predictive performance that is best to the training images to generate the predicted facial action units for each training image;
for each avatar, calculating an avatar-specific predictive performance of the instance of the machine learning model having the predictive performance that is best based on the predicted and ground truth facial action units for the testing avatar;
identifying a first set of features common to the avatars for which the avatar-specific predictive performance was better than a first threshold;
identifying a second set of features common to the avatars for which the avatar-specific predictive performance was worse than a second threshold;
identifying the features present only in the second set, as different features; and
generating new avatars having the difference features; and
retraining the instances of the machine learning model using the new avatars.

### Claim 12

A system comprising:
a head-mountable display (HMD) having one or multiple cameras to capture a set of facial images of a wearer of the HMD;
a processor; and
a memory storing program code executable by the processor to:
preprocess the facial images so that the images better resemble synthetically rendered images;
applying a machine learning model to the preprocessed facial images to generate predicted wearer facial action units for a facial expression of the wearer exhibited within the facial images;
postprocessing the predicted wearer facial action units to smooth the predicted wearer facial action units; and
retarget the postprocessed predicted facial action units onto an avatar corresponding to the wearer to render the avatar with the facial expression of the wearer for display.

### Claim 13 (depends on 12)

The system of claim 12, wherein the processor is to preprocess the facial images by performing adaptive histogram equalization,
and wherein the processor is to postprocess the predicted wearer facial action units by performing average mean filtering.

### Claim 14 (depends on 12)

The system of claim 12, wherein the machine learning model is initially trained using a plurality of avatars, and is retrained using new avatars having features specific to the avatars for which predictive performance of the initially trained machine learning model is worse than a threshold.

### Claim 15 (depends on 12)

The system of claim 12, wherein the machine learning model that is applied is an instance of a plurality of instances of the machine learning model for which predictive performance was best, each instance corresponding to different training parameters.
