# US 12,003,883 B2 · Facial synthesis in overlaid augmented reality content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,883 B2 - Document ID: US12003883B2 - Status: Granted - Kind: B2 - Application number: 17/706,035 - Filed: 2022-03-28 - Published: 2022-10-06 - Granted: 2024-06-04 - Earliest claimed: 2021-03-31 - Assignees: Snap Inc. - Applicants: Snap Inc. - Inventors: Nikita Demidov; Roman Golobokov; Alina Melnyk; Jeremy Baker Voss; Aleksei Bromot - Web page: https://patentgrep.com/patent/US12003883B2 ## CPC - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/60: Creating or editing images; Combining images with text - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 10/82: using neural networks - G06V 40/166: {using acquisition arrangements} - H04L 51/21: Monitoring or handling of messages - H04L 51/52: for supporting social networking services - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - H04N 2005/2726: {for simulating a person's appearance, e.g. hair style, glasses, clothes} - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The subject technology receives at least one signal from a computing device, the at least one signal comprising at least one of a current time, battery power, sensor information, or location information. The subject technology generates a digital sticker, the digital sticker including graphical content indicating information based at least in part based on the at least one signal and media content including an image of a target face, the image of the target face being modified based on at least one of sets of source pose parameters to mimic at least one of positions of a head of a source actor and at least one of facial expressions of the source actor. The subject technology provides augmented reality content for display on a computing device, the augmented reality content including the digital sticker as an overlay on at least a portion of the augmented reality content. ## Claims ### Claim 1 A method, comprising: receiving, by one or more hardware processors, at least one signal from a computing device, the at least one signal comprising at least one of a current time, battery power, sensor information, or location information; generating, by the one or more hardware processors, a digital sticker, the digital sticker including graphical content indicating information based at least in part based on the at least one signal and media content including an image of a target face, the image of the target face being modified based on at least one of sets of source pose parameters to mimic at least one of positions of a head of a source actor and at least one of facial expressions of the source actor; and providing, by the one or more hardware processors, augmented reality content for display on a computing device, the augmented reality content including the digital sticker as an overlay on at least a portion of the augmented reality content. ### Claim 2 (depends on 1) The method of claim 1, further comprising: sending the augmented reality content including the digital sticker as a message to a user in a messaging system. ### Claim 3 (depends on 1) The method of claim 1, wherein generating the digital sticker comprises: performing a lookup in an augmented reality (AR) image database based at least in part on the at least one signal from the computing device, wherein the AR image database comprises a database storing a set of images for including in AR content; determining that first metadata associated with a particular image from the set of images matches second metadata associated with the at least one signal, the first metadata comprising information indicating a type of signal corresponding to the current time, battery power, sensor information, or location information, the second metadata comprising information indicating that the particular image is related to the current time, battery power, sensor information, or location information; and selecting the particular image for generating, in part, the digital sticker. ### Claim 4 (depends on 3) The method of claim 3, further comprising: determining a first set of dimensions associated with the digital sticker, the set of dimensions including at least a height of a set of pixels of the digital sticker and a width of the set of pixels of the digital sticker; selecting an area of interest within the set of dimensions; and generating the digital sticker including an image in the area of interest, wherein the image includes a representation of the current time, battery power, sensor information, or location information. ### Claim 5 (depends on 4) The method of claim 4, wherein providing the augmented reality content for display comprises: determining an anchor point in a current view of a camera of the computing device, the current view representing a three-dimensional scene captured by the camera at the current time, the anchor point corresponding to X, Y, and Z coordinates (x, y, z) at a distance from the camera of the computing device and within the three-dimensional scene; and rendering the augmented reality content based on the anchor point in the three-dimensional scene, the augmented reality content including the digital sticker. ### Claim 6 (depends on 5) The method of claim 5, wherein rendering the augmented reality content at the anchor point in the three-dimensional scene comprises: determining a set of dimensions associated with the augmented reality content, the set of dimensions including at least a height and width of the digital sticker; and rendering the digital sticker at a distance that is offset from the anchor point based on a percentage of at least the height or the width of the digital sticker. ### Claim 7 (depends on 3) The method of claim 3, wherein the first metadata associated with the particular image further comprises a description, a length of time for displaying the particular image, a set of tags associated with the particular image, a title of the particular image, information related to an author of the particular image, and a set of categories associated with the particular image, and wherein the second metadata associated with the at least one signal further comprises raw data associated with the at least one signal, the raw data comprising at least a set of numerical values provided by a sensor or signal provider of the computing device. ### Claim 8 (depends on 1) The method of claim 1, wherein the image of the target face being modified based on the at least one of sets of source pose parameters is based on generating, using a first encoder network, a first set of facial features based on the positions of the head of the source actor and the at least one of facial expressions of the source actor, the generating producing image data of lower dimensionality than image data from source media content, wherein the lower dimensionality comprises a lower resolution than a particular resolution of the image data of the source media content. ### Claim 9 (depends on 8) The method of claim 8, further comprising: generating, using a first decoder network associated with the target face, a first output image, the first output image comprising a modification of a representations of the head of the source actor and the at least one of facial expressions of the source actor based on a set of facial features from the target face. ### Claim 10 (depends on 9) The method of claim 9, wherein the first encoder network comprises a first deep convolutional neural network, the first decoder network comprises a second deep convolutional neural network, and the first deep convolutional neural network and the second deep convolutional neural network are different neural networks. ### Claim 11 A system comprising: a processor; and a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising: receiving, by one or more hardware processors, at least one signal from a computing device, the at least one signal comprising at least one of a current time, battery power, sensor information, or location information; generating, by the one or more hardware processors, a digital sticker, the digital sticker including graphical content indicating information based at least in part based on the at least one signal and media content including an image of a target face, the image of the target face being modified based on at least one of sets of source pose parameters to mimic at least one of positions of a head of a source actor and at least one of facial expressions of the source actor; and providing, by the one or more hardware processors, augmented reality content for display on a computing device, the augmented reality content including the digital sticker as an overlay on at least a portion of the augmented reality content. ### Claim 12 (depends on 11) The system of claim 11, wherein the operations further comprise: sending the augmented reality content including the digital sticker as a message to a user in a messaging system. ### Claim 13 (depends on 11) The system of claim 11, wherein generating the digital sticker comprises: performing a lookup in an augmented reality (AR) image database based at least in part on the at least one signal from the computing device, wherein the AR image database comprises a database storing a set of images for including in AR content; determining that first metadata associated with a particular image from the set of images matches second metadata associated with the at least one signal, the first metadata comprising information indicating a type of signal corresponding to the current time, battery power, sensor information, or location information, the second metadata comprising information indicating that the particular image is related to the current time, battery power, sensor information, or location information; and selecting the particular image for generating, in part, the digital sticker. ### Claim 14 (depends on 13) The system of claim 13, wherein the operations further comprise: determining a first set of dimensions associated with the digital sticker, the set of dimensions including at least a height of a set of pixels of the digital sticker and a width of the set of pixels of the digital sticker; selecting an area of interest within the set of dimensions; and generating the digital sticker including an image in the area of interest, wherein the image includes a representation of the current time, battery power, sensor information, or location information. ### Claim 15 (depends on 14) The system of claim 14, wherein providing the augmented reality content for display comprises: determining an anchor point in a current view of a camera of the computing device, the current view representing a three-dimensional scene captured by the camera at the current time, the anchor point corresponding to X, Y, and Z coordinates (x, y, z) at a distance from the camera of the computing device and within the three-dimensional scene; and rendering the augmented reality content based on the anchor point in the three-dimensional scene, the augmented reality content including the digital sticker. ### Claim 16 (depends on 15) The system of claim 15, wherein rendering the augmented reality content at the anchor point in the three-dimensional scene comprises: determining a set of dimensions associated with the augmented reality content, the set of dimensions including at least a height and width of the digital sticker; and rendering the digital sticker at a distance that is offset from the anchor point based on a percentage of at least the height or the width of the digital sticker. ### Claim 17 (depends on 13) The system of claim 13, wherein the first metadata associated with the particular image further comprises a description, a length of time for displaying the particular image, a set of tags associated with the particular image, a title of the particular image, information related to an author of the particular image, and a set of categories associated with the particular image, and wherein the second metadata associated with the at least one signal further comprises raw data associated with the at least one signal, the raw data comprising at least a set of numerical values provided by a sensor or signal provider of the computing device. ### Claim 18 (depends on 11) The system of claim 11, wherein the image of the target face being modified based on the at least one of sets of source pose parameters is based on generating, using a first encoder network, a first set of facial features based on the positions of the head of the source actor and the at least one of facial expressions of the source actor, the generating producing image data of lower dimensionality than image data from source media content, wherein the lower dimensionality comprises a lower resolution than a particular resolution of the image data of the source media content. ### Claim 19 (depends on 18) The system of claim 18, wherein the operations further comprise: generating, using a first decoder network associated with the target face, a first output image, the first output image comprising a modification of a representations of the head of the source actor and the at least one of facial expressions of the source actor based on a set of facial features from the target face. ### Claim 20 A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising: receiving, by one or more hardware processors, at least one signal from a computing device, the at least one signal comprising at least one of a current time, battery power, sensor information, or location information; generating, by the one or more hardware processors, a digital sticker, the digital sticker including graphical content indicating information based at least in part based on the at least one signal and media content including an image of a target face, the image of the target face being modified based on at least one of sets of source pose parameters to mimic at least one of positions of a head of a source actor and at least one of facial expressions of the source actor; and providing, by the one or more hardware processors, augmented reality content for display on a computing device, the augmented reality content including the digital sticker as an overlay on at least a portion of the augmented reality content.