# US 12,002,139 B2 · Robust facial animation from video using neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,139 B2 - Document ID: US12002139B2 - Status: Granted - Kind: B2 - Application number: 17/677,123 - Filed: 2022-02-22 - Published: 2022-08-25 - Granted: 2024-06-04 - Earliest claimed: 2021-02-22 - Assignees: Roblox Corporation - Applicants: Roblox Corporation - Inventors: Inaki Navarro; Dario Kneubuhler; Tijmen Verhulsdonck; Eloi Du Bois; Will Welch; Vivek Verma; Ian Sachs; Kiran Bhat - Web page: https://patentgrep.com/patent/US12002139B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06T 7/73: using feature-based methods - G06V 10/24: Aligning, centring, orientation detection or correction of the image - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/82: using neural networks - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - G06V 40/176: {Dynamic expression} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30201: Face - G06T 2215/16: Using real world measurements to influence rendering - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Implementations described herein relate to methods, systems, and computer-readable media to generate animations for a 3D avatar from input video captured at a client device. A camera may capture video of a face while a trained face detection model and a trained regression model output a set of FACS weights, head poses, and facial landmarks to be translated into the animations of the 3D avatar. Additionally, a higher level-of-detail may be intelligently selected based upon user preferences and/or computing conditions at the client device. ## Claims ### Claim 1 A computer-implemented method, comprising: identifying, using a fully convolutional network, a set of bounding box candidates from a first frame of a video, wherein each bounding box candidate includes a face; refining, using a convolutional neural network, the set of bounding box candidates into a bounding box; obtaining a first set of one or more of a predefined facial expression weight, a head pose, and facial landmarks based on the bounding box and the first frame using an overloaded output convolutional neural network; generating a first animation frame of an animation of a three-dimensional (3D) avatar based on the first set of the one or more of the predefined facial expression weight, the head pose, and the facial landmarks, wherein a head pose of the avatar matches the head pose in the first set and facial landmarks of the avatar match the facial landmarks in the first set; and for each additional frame of the video subsequent to the first frame, detecting whether the bounding box, applied to the additional frame, includes the face; if it is detected that the bounding box includes the face, bypassing the fully convolutional network and the convolutional neural network, and obtaining an additional set of the one or more predefined facial expression weights, head poses, and facial landmarks based on the bounding box and the additional frame, by using the overloaded output convolutional neural network; and generating an additional animation frame of the animation of the 3D avatar using the additional set. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein, if it is detected that the bounding box for the additional frame does not include the face, setting the additional frame as the first frame. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising identifying, by the overloaded output convolutional neural network, a tongue out condition wherein a tongue is detected in the face. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, wherein identifying the tongue out condition is based on sub-model of the overloaded output convolutional neural network. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the overloaded output convolutional network is a first neural network, the method further comprising refining the first set of one or more of the predefined facial expression weight, the head pose, and the facial landmarks based on the bounding box and the first frame using an additional overloaded output convolutional neural network. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the additional overloaded output convolutional neural network provides a higher level of detail as compared to the first neural network. ### Claim 7 (depends on 5) The computer-implemented method of claim 5, further comprising identifying, by the additional overloaded output convolutional neural network, a tongue out condition wherein a tongue is detected in the face. ### Claim 8 (depends on 5) The computer-implemented method of claim 5, wherein an input frame resolution of the first neural network is less than an input frame resolution of the additional overloaded output convolutional neural network. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the overloaded output convolutional neural network is trained on a set of artificially generated input frames of an artificial video. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein the overloaded output convolutional neural network is further trained with a set of real images that are hand-labeled. ### Claim 11 A system comprising: a memory with instructions stored thereon; and a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform operations comprising: identifying, using a fully convolutional network, a set of bounding box candidates from a first frame of a video, wherein each bounding box candidate includes a face; refining, using a convolutional neural network, the set of bounding box candidates into a bounding box; obtaining a first set of one or more of a predefined facial expression weight, a head pose, and facial landmarks based on the bounding box and the first frame using an overloaded output convolutional neural network; generating a first animation frame of an animation of a three-dimensional (3D) avatar based on the first set of the one or more of the predefined facial expression weight, the head pose, and the facial landmarks, wherein a head pose of the avatar matches the head pose in the first set and facial landmarks of the avatar match the facial landmarks in the first set; and for each additional frame of the video subsequent to the first frame, detecting whether the bounding box, applied to the additional frame, includes the face; if it is detected that the bounding box includes the face, bypassing the fully convolutional network and the convolutional neural network, and obtaining an additional set of the one or more predefined facial expression weights, head poses, and facial landmarks based on the bounding box and the additional frame, by using the overloaded output convolutional neural network; and generating an additional animation frame of the animation of the 3D avatar using the additional set. ### Claim 12 (depends on 11) The system of claim 11, wherein, if it is detected that the bounding box for the additional frame does not include the face, setting the additional frame as the first frame. ### Claim 13 (depends on 11) The system of claim 11, wherein the operations further comprise identifying, by the overloaded output convolutional neural network, a tongue out condition wherein a tongue is detected in the face. ### Claim 14 (depends on 11) The system of claim 11, wherein the overloaded output convolutional network is a first neural network, the operations further comprising additionally refining the first set of one or more of the predefined facial expression weight, the head pose, and facial landmarks based on the bounding box and the first frame using an additional overloaded output convolutional neural network. ### Claim 15 (depends on 14) The system of claim 14, wherein the additional overloaded output convolutional neural network provides a higher level of detail as compared to the first neural network. ### Claim 16 A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, causes the processing device to perform operations comprising: identifying, using a fully convolutional network, a set of bounding box candidates from a first frame of a video, wherein each bounding box candidate includes a face; refining, using a convolutional neural network, the set of bounding box candidates into a bounding box; obtaining a first set of one or more of a predefined facial expression weight, a head pose, and facial landmarks based on the bounding box and the first frame using an overloaded output convolutional neural network; generating a first animation frame of an animation of a three dimensional (3D) avatar based on the first set of the one or more of the predefined facial expression weight, the head pose, and the facial landmarks, wherein a head pose of the avatar matches the head pose in the first set and facial landmarks of the avatar match the facial landmarks in the first set; and for each additional frame of the video subsequent to the first frame, detecting whether the bounding box, applied to the additional frame, includes the face; if it is detected that the bounding box includes the face, bypassing the fully convolutional network and the convolutional neural network, and obtaining an additional set of the one or more predefined facial expression weights, head poses, and facial landmarks based on the bounding box and the additional frame, by using the overloaded output convolutional neural network; and generating an additional animation frame of the animation of the 3D avatar using the additional set. ### Claim 17 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein, if it is detected that the bounding box for the additional frame does not include the face, setting the additional frame as the first frame. ### Claim 18 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the operations further comprise identifying, by the overloaded output convolutional neural network, a tongue out condition wherein a tongue is detected in the face. ### Claim 19 (depends on 16) The non-transitory computer-readable medium of claim 16, wherein the overloaded output convolutional network is a first neural network, the operations further comprising additionally refining the first set of one or more of the predefined facial expression weight, the head pose, and facial landmarks based on the bounding box and the first frame using an additional overloaded output convolutional neural network. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the additional overloaded output convolutional neural network provides a higher level of detail as compared to the first neural network.