# US 12,602,855 B2 · Live model prompting and real-time output of photoreal synthetic content > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,855 B2 - Document ID: US12602855B2 - Status: Granted - Kind: B2 - Application number: 18/484,586 - Filed: 2023-10-11 - Published: 2024-10-17 - Granted: 2026-04-14 - Earliest claimed: 2023-04-14 - Assignees: Metaphysic Limited - Applicants: Metaphysic Limited - Inventors: Thomas Graham; Chris Ume; Jo Plaete; Martin Adams - Web page: https://patentgrep.com/patent/US12602855B2 ## CPC - G06T 13/40: of characters, e.g. humans, animals or virtual beings - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/08: Learning methods - G06N 20/00: Machine learning - G06T 13/205: {driven by audio data} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Prompting a trained artificial intelligence (AI) model(s) to output photoreal synthetic content in real-time is described. In some examples, one or more AI models are trained using sequential video frames as training data to obtain one or more trained AI models configured to generate temporally-coherent output data. In an example process, user-provided prompt data representing a prompt provided by a user is received, output data representing synthetic content is generated using the trained AI model(s) based at least in part on the user-provided prompt data, and video content featuring the synthetic content is caused to be displayed on a display based at least in part on the output data. In some examples, the output data is provided to the trained AI model(s) as part of a feedback loop to generate further output data as part of a real-time, iterative prompting system. ## Claims ### Claim 1 A method comprising: receiving, by one or more processors, input video data generated by a video capture device as the video capture device is capturing a real-world scene, the input video data representing a face of a subject in the real-world scene; receiving, by the one or more processors, user-provided prompt data representing a voice prompt to make the subject look younger; generating, by the one or more processors, using one or more trained machine learning models based at least in part on the user-provided prompt data, and using frames of the input video data representing the face of the subject, output data representing a synthetic face of the subject that looks younger than the face of the subject exhibited in the input video data; and causing, by the one or more processors, based at least in part on the output data, video content featuring the synthetic face of the subject overlaid on the face of the subject exhibited in the input video data to be displayed, in real-time as the video capture device is capturing the real-world scene, on a display that is viewable by a user who provided the voice prompt in order to de-age the subject live, while the video capture device is capturing the real-world scene. ### Claim 2 (depends on 1) The method of claim 1, further comprising: generating, by the one or more processors, output video data corresponding to the video content by overlaying the synthetic face of the subject onto the face of the subject in the input video data, wherein the causing the video content to be displayed on the display is based at least in part on the output video data. ### Claim 3 (depends on 1) The method of claim 1, further comprising: providing, by the one or more processors, face data representing the face of the subject to the one or more trained machine learning models as additional prompt data, wherein the generating the output data using the one or more trained machine learning models is further based on the additional prompt data. ### Claim 4 (depends on 1) The method of claim 1, wherein: the user-provided prompt data comprises audio data representing the voice prompt provided by the user and captured by a microphone; the method further comprises: generating, by the one or more processors, based at least in part on the audio data, text data representing the voice prompt; and identifying, by the one or more processors, from a list of predefined text prompts, a predefined text prompt that corresponds to the text data; and the generating the output data is based at least in part on the predefined text prompt. ### Claim 5 (depends on 1) The method of claim 1, further comprising: receiving, by the one or more processors, second user-provided prompt data representing a second prompt provided by the user; generating, by the one or more processors, using the one or more trained machine learning models based at least in part on the second user-provided prompt data, second output data representing synthetic background content; and causing, by the one or more processors, based at least in part on the output data and the second output data, second video content featuring the synthetic face of the subject and the synthetic background content to be displayed, in real-time as the video capture device is capturing the real-world scene, on the display. ### Claim 6 A method comprising: receiving, by one or more processors, input video data generated by a video capture device as the video capture device is capturing a real-world scene, the input video data representing a face of a subject in the real-world scene; receiving, by the one or more processors, user-provided prompt data representing a prompt to make the subject look younger; generating, by the one or more processors, using one or more trained machine learning models based at least in part on the user-provided prompt data, and using frames of the input video data representing the face of the subject, output data representing a synthetic face of the subject that looks younger than the face of the subject; and causing, by the one or more processors, based at least in part on the output data, video content featuring the synthetic face of the subject overlaid on the face of the subject exhibited in the input video data to be displayed, in real-time as the video capture device is capturing the real-world scene, on a display that is viewable by a user who provided the prompt. ### Claim 7 (depends on 6) The method of claim 6, wherein the output data is first output data, the synthetic face is a first synthetic face, the video content is first video content, and the method further comprises: providing, by the one or more processors, as part of a feedback loop, the first output data to the one or more trained machine learning models; generating, by the one or more processors, using the one or more trained machine learning models based at least in part on the first output data, second output data representing a second synthetic face of the subject that looks younger than the face of the subject; and causing, by the one or more processors, based at least in part on the second output data, second video content featuring the second synthetic face of the subject overlaid on the face of the subject exhibited in the input video data to be displayed, in real-time as the video capture device is capturing the real-world scene, on the display. ### Claim 8 (depends on 6) The method of claim 6, further comprising: generating, by the one or more processors, output video data corresponding to the video content by overlaying the synthetic face of the subject onto the face of the subject in the input video data, wherein the causing the video content to be displayed on the display is based at least in part on the output video data. ### Claim 9 (depends on 5) The method of claim 5, further comprising: providing, by the one or more processors, face data representing the face of the subject to the one or more trained machine learning models as additional prompt data, wherein the generating the output data using the one or more trained machine learning models is further based on the additional prompt data. ### Claim 10 (depends on 8) The method of claim 8, wherein: the output video data comprises a series of frames, and the output data is generated for a subset, but not all, of the series of frames. ### Claim 11 (depends on 6) The method of claim 6, wherein: the user-provided prompt data comprises audio data representing a voice prompt provided by the user and captured by a microphone; the method further comprises: generating, by the one or more processors, based at least in part on the audio data, text data representing the voice prompt; and identifying, by the one or more processors, from a list of predefined text prompts, a predefined text prompt that corresponds to the text data; and the generating the output data is based at least in part on the predefined text prompt. ### Claim 12 (depends on 6) The method of claim 6, further comprising constraining the one or more trained machine learning models to a predefined set of prompts, wherein the prompt provided by the user is a predefined prompt of the predefined set of prompts. ### Claim 13 (depends on 6) The method of claim 6, further comprising: receiving, by the one or more processors, second user-provided prompt data representing a second prompt provided by the user; generating, by the one or more processors, using the one or more trained machine learning models based at least in part on the second user-provided prompt data, second output data representing synthetic background content; and causing, by the one or more processors, based at least in part on the output data and the second output data, second video content featuring the synthetic face of the subject and the synthetic background content to be displayed, in real-time as the video capture device is capturing the real-world scene, on the display. ### Claim 14 A system comprising: one or more processors; and memory storing computer-executable instructions that, when executed by the one or more processors, cause performance of operations comprising: receiving input video data generated by a video capture device as the video capture device is capturing a real-world scene, the input video data representing a face of a subject in the real-world scene; receiving user-provided prompt data representing a prompt to make the subject look younger; generating, using one or more trained machine learning models based at least in part on the user-provided prompt data, and using frames of the input video data representing the face of the subject, output data representing a synthetic face of the subject that looks younger than the face of the subject; and causing, based at least in part on the output data, video content featuring the synthetic face of the subject overlaid on the face of the subject exhibited in the input video data to be displayed, in real-time as the video capture device is capturing the real-world scene, on a display that is viewable by a user who provided the prompt. ### Claim 15 (depends on 14) The system of claim 14, the operations further comprising: generating output video data corresponding to the video content by overlaying the synthetic face of the subject onto the face of the subject in the input video data, wherein the causing the video content to be displayed on the display is based at least in part on the output video data. ### Claim 16 (depends on 15) The system of claim 15, wherein: the output video data comprises a series of frames, and the output data is generated for a subset, but not all, of the series of frames. ### Claim 17 (depends on 14) The system of claim 14, the operations further comprising: providing face data representing the face of the subject to the one or more trained machine learning models as additional prompt data, wherein the generating the output data using the one or more trained machine learning models is further based on the additional prompt data. ### Claim 18 (depends on 14) The system of claim 14, the operations further comprising constraining the one or more trained machine learning models to a predefined set of prompts, wherein the prompt provided by the user is a predefined prompt of the predefined set of prompts. ### Claim 19 (depends on 14) The system of claim 14, wherein: the user-provided prompt data comprises audio data representing a voice prompt provided by the user and captured by a microphone; the operations further comprise: generating, based at least in part on the audio data, text data representing the voice prompt; and identifying, from a list of predefined text prompts, a predefined text prompt that corresponds to the text data; and the generating the output data is based at least in part on the predefined text prompt. ### Claim 20 (depends on 14) The system of claim 14, wherein the output data is first output data, the synthetic face is a first synthetic face, the video content is first video content, and the operations further comprise: providing, as part of a feedback loop, the first output data to the one or more trained machine learning models; generating, using the one or more trained machine learning models based at least in part on the first output data, second output data representing a second synthetic face of the subject that looks younger than the face of the subject; and causing, based at least in part on the second output data, second video content featuring the second synthetic face of the subject overlaid on the face of the subject exhibited in the input video data to be displayed, in real-time as the video capture device is capturing the real-world scene, on the display. ### Claim 21 (depends on 14) The system of claim 14, the operations further comprising: receiving second user-provided prompt data representing a second prompt provided by the user; generating, using the one or more trained machine learning models based at least in part on the second user-provided prompt data, second output data representing synthetic background content; and causing, based at least in part on the output data and the second output data, second video content featuring the synthetic face of the subject and the synthetic background content to be displayed, in real-time as the video capture device is capturing the real-world scene, on the display.